The Challenge
Optimising data discovery across extensive national statistics
The Singapore Department of Statistics (DOS) manages the SingStat Table Builder, a comprehensive platform hosting some 2,400 data tables from 70 public sector agencies. This extensive repository provides detailed insights into Singapore’s economic and socio-demographic landscape.
To further enhance user experience and data accessibility, DOS explored innovative approaches to allow users to interact with national statistical repositories through conversational AI technology. This initiative eliminates traditional barriers by removing the need for users to be familiar with technical jargon or specific dataset names, while also addressing the time-consuming process of searching through similarly named tables to find the required information. By simplifying access to statistical information, DOS aims to foster data-driven decision-making across all user groups and improve productivity in data sourcing.
Our Approach
Breathing life into data with AI — building real capability
Working closely with DOS, the team adopted a systematic, design-led approach across four stages to develop an innovative solution for statistical data access.
- Discovery: uncovering the real user need: We started by understanding user search behaviour through analysis of website search logs and conducting stakeholder interviews. Key findings indicated that users typically formulate queries using natural language and conceptual terms instead of technical terms or table titles. This insight became the cornerstone of our strategy.
- Design: crafting a human-centred AI: Armed with this understanding, we designed a human-centred solution that puts user intent first. We built an AI tool that could understand natural language and match the semantic meaning of a user's question to the most relevant data, facilitating intuitive data discovery.
- Detailing: bridging design and technology: We bridged the gap between the interface design and the back-end AI. Our team modelled the rich metadata from some 1,000 time series datasets and converted them into vector embeddings, allowing the AI to understand the datasets. We worked in close collaboration with DOS’s domain experts throughout this process, ensuring the AI model was fast, reliable and accurate.
- Development: delivering a complete, end-to-end solution: Our interdisciplinary team engineered the complete system from the ground up. This involved building the intuitive, mobile-responsive chat interface, developing the complex AI logic, and deploying the entire system on a secure and scalable AWS cloud infrastructure. Our co-teaming model ensured the final product was built with the client, not just for them.
The Outcomes
Democratising data and empowering a culture of innovation
The launch of the Statistics And Data Retrieval A.I. assistant (SANDRA) introduced conversational AI capabilities to SingStat Table Builder to enhance statistical data discovery. This allows users to locate relevant datasets using natural language interactions.
- Effortless data discovery: The chatbot enables users to discover statistical data using natural language queries, removing the need for technical terminology. Users can now ask questions in plain English and instantly view the relevant data in an interactive chart or table. The AI also intelligently recommends other related datasets which may be of interest to users, enabling a deeper exploration of topics.
- Improved operational efficiency: With the chatbot assisting with data enquiries from public users, this allows DOS officers to dedicate their time to complex queries and higher-value work.
- A future-proof investment: The improved data discoverability allowed users to independently navigate and extract required data. By making detailed statistics more accessible to a wider audience—from researchers to members of public—we ensured DOS’s investment delivered on its promise. The resulting increase in data utilisation, demonstrated that when data is easy to find, its power to inform and drive progress will be further realised.
The final product, a public-facing and easy-to-use tool, successfully democratised access to one of the nation's most important data repositories, transforming it into a dynamic and conversational resource.
