The data that matter - a new approach to critical data sources for core statistics

The Office for National Statistics (ONS) is committed to producing trustworthy, independent, high-quality statistics that underpin the UK's most critical economic and societal decisions and inform the public. The evolving data landscape, shaped by technological advancements and changing user expectations, means the ONS must continuously adapt its approach to data usage and sourcing.

The ONS has access to one of the most extensive and valuable data estates in the public sector, and there remains significant opportunity to further harness the breadth and depth of those data sources to support core statistics. While significant progress has been made in acquiring, understanding, integrating, and managing diverse datasets, substantial untapped potential remains – both in fixing the data foundations and maximising the analytical use of existing data in our production of statistics.

The Office for Statistics Regulation (OSR) recommended in its report Systemic Review of ONS Economic Statistics: April 2025 that:

"in addition to the recovery plan and drawing on the overall strategy for economic statistics, Office for National Statistics (ONS) should develop and publish a regularly updated vision and strategy for the data sources used to compile its economic statistics. This publication should include a "roadmap" setting out how the use of surveys and administrative (and other non-survey-based) sources will be developed in an integrated way, including the development of methods that combine data sources, as well as any barriers that ONS foresees and the support it needs from others to address them."

The new approach explained herein, published alongside the first quarterly ONS plan for economic statistics: progress update, responds to this request, and sets out the principles, priorities, and practical approaches underpinning the ONS's work to source and exploit the data that matters for core statistics.

This approach will be reviewed regularly, informed by feedback from analysts, data providers, and users, to ensure the ONS remains at the forefront of public sector data sourcing in the UK, and updates will be included as part of the quarterly ONS plan for economic statistics: progress updates, where appropriate.

Keeping data secure and confidential is the ONS's first priority. Our website explains how we keep data safe and protect personal information so that no one can be identified from the statistics we publish.

Directly collected and indirectly acquired data sources

The Office for National Statistics (ONS) generates, collects and acquires a large volume of data. This includes survey data collected from individuals and businesses, administrative data acquired from public sources, and alternative data such as financial transaction data or retail point of sale data.

The usages of this data by the ONS are varied and include supporting methodological and statistical research; as part of survey operations, for example, providing sampling frames and weighting approaches; and directly feeding into official statistics.

See Section 8: Annex A, Sourcing methods for the description of our three methods of data sourcing:

• direct data collection

• indirect administrative data collection from other government sources

• indirect alternative data collection from non-government sources

Legislative context

The ONS has broad legal powers to both collect and acquire data. The ONS, as the executive office of the Statistics Board (UK Statistics Authority), is created and bound by the Statistics and Registration Service Act 2007 (SRSA). When passed in 2007, the SRSA provided the ONS with a general power to undertake any such voluntary statistical surveys as it deemed necessary to meet its statutory function alongside a limited number of permissive data sharing powers.

The Digital Economy Act 2017 (DEA) amended the SRSA, giving the ONS much greater data acquisition powers. These included a permissive power for any public authority to share the data it holds with the ONS, over-riding any existing statute bar or obligation of confidence other than those arising from data protection legislation, and a mandatory power which the ONS can use to serve notice on both public and private bodies to require the disclosure of data, subject to certain safeguards.

The SRSA is supplemented by both the Census Act 1920, under which the ONS can take the decennial census of England and Wales, and the Statistics of Trade Act 1947, which provides a statutory power for the ONS to undertake mandatory surveys of UK businesses.

There are three strategic objectives supporting the vision "to source and exploit the data that matters for Office for National Statistics (ONS) core outputs - ensuring the ongoing value, reliability, and inclusivity of our national statistics":

1. To manage and improve the quality of our critical data sources

2. To maximise value from existing data sources

3. To deliver a transparent and coherent approach to sourcing new data

The current landscape

The Office for National Statistics (ONS) holds over 2,500 data assets from various sources. Over 2,000 of these data assets are used in some way to support economic and population statistics, with approximately 440 sources contributing to gross domestic product (GDP) statistics alone. Not only does this illustrate the volume of data sources that contribute to compiling, verifying and validating the quality of ONS statistics, but it also points to the complexity and dependencies involved in doing so.

Most of the survey sources are designed and collected by the ONS to support core statistics. However, most of the non-survey sources are produced and supplied by third parties, where the ONS relies on regular data deliveries to support the production of core statistics.

Figure 1 shows that such data deliveries can vary in their regularity, from bi-annual supplies through to daily, which, taken together, provide a rich picture of our population, economy and society. There are varying levels of maturity of these supplies, ranging from automated and reliable deliveries, to manual supplies with recurring operational or quality issues, such as late delivery, poor metadata, or missing files.

Figure 1: How often administrative data sources support ONS core statistics

Table diagram showing regularity of administrative data deliveries supporting ONS core statistics 

Notes:

1. Includes regular data deliveries managed by the Data Growth and Operations Directorate, it excludes regular data deliveries managed directly by other business areas.

In terms of our ability to share data across government, we have identified a first version of the ONS's essential shared data assets (ESDAs), which represent the top 20 microdata assets shared with external users. This list will be regularly updated, as prioritisation and context changes. These are almost all survey sources. These are included at Annex B – first version of ONS essential shared data assets for external sharing and illustrate the importance of the ONS as a data producer for both decision-making and informing society. However, we have not undertaken a systematic, comprehensive review of the critical data which underpin our core statistics.

Understanding the relative importance of data sources to each core output is essential for maintaining reliable end-to-end data journeys and using data of known quality. Knowledge of end-to-end production systems often relies on individual expertise and quality assurance happens at a later stage of processing. This can lead to single points of failure, and on occasion analysts outside of the ONS will support by highlighting data anomalies after publication.

Critical business processes have grown organically and at pace to meet demands, with independent sourcing catalogues, tables and metadata repositories being developed to meet the imminent need of a business area. To fully understand and exploit the data across the organisation, there is a need to bring greater oversight to recording and then sharing quality information about our data sources. Whilst ONS's Information and Data Assets Register provides a source of truth of the data we hold, there are additional tools we can now deploy to evolve and expand how users find, discover, understand, and exploit the data sources the ONS has access to.

The opportunity

Through understanding our critical data sources, we can invest efforts into focused management of these sources, ensuring stability and quality is managed through the end-to-end data lifecycle. This will directly support improvements to quality and reliability of our core outputs, helping to identify, contextualise and mitigate quality risks.

To strengthen this focus on quality management we are introducing new, consolidated data ownership roles to ensure full value is derived from our sources, including stronger accountability and investment in managing their quality. The new roles are part of a wider initiative to introduce a new Data Governance Framework for the ONS that will see improvements made in the way in which we are working with data.

Alongside this, we will invest in a tooling suite, which enables effective metadata usability across the organisation, and which is foundational for the organisation to benefit from the operational efficiencies and quality improvements that technology can offer.

This work focuses on the better management of our data sources and sits alongside the Survey Improvement and Enhancement Plan (SIEP), which covers improvements to survey collection, methods and processing.

Our commitments

We will define our critical data for core statistics

We have developed criteria for defining what makes a data source critical, including whether a given data source feeds into a core output alongside whether the data source is unique, with no viable alternative to deliver against the requirements of an output.

Work is under way to pilot this criticality criteria against core outputs ahead of broader implementation across all data domains. This has supported identification and understanding of the critical data sources supporting admin-based population estimates (ABPEs) and enabled investment in improving the quality and end-to-end management of the most critical data sources – see Case study 1.

We will take an end-to-end approach to managing our critical data sources, ensuring processes are robust and resilient, and improving data quality

Having identified our critical data, we will prioritise efforts, relationship management, and investment into ensuring these sources are reliable, to an agreed quality standard, and optimised for the production of core statistics and re-use across the ONS.

We will prioritise appointing data owners and stewards for our critical data, to ensure clearer responsibility and accountability for managing the quality and availability of data. A newly established Data Governance Office will work with respective data stewards and owners to implement Data Quality Action Plans for all critical data sources. This will include key performance indicators to support the measurement of data quality and maturity, alongside the identification of risks and issues.

We will monitor the maturity of administrative data pipelines supporting our critical data sources and work with partners to address quality issues. We will automate supplies wherever appropriate to ensure greater reliability and stability of regular data deliveries.

We will develop a strategic solution to support greater data observability, reuse, and access

We will develop a common metamodel, setting the minimum metadata requirements for our data. We will work with data owners to enhance existing metadata, starting with critical data sources, to align with these standards and ensure that new data sources comply with minimum metadata requirements.

We will implement a tooling suite to provide a repository for this metadata and a discovery layer to our data, enabling ONS analysts to search intuitively for the data that matters for their analysis and statistical production, and supporting greater transparency around the usage of data for external stakeholders and the public.

The current landscape

The Office for National Statistics (ONS)'s technical landscape has evolved organically over many years, with data collected, stored, and processed on multiple services and platforms.

The existing architecture and federation of technology systems creates a complex data hosting and statistical processing environment, which can create barriers for access and reuse of data, with users facing long lead times to ingest, access and use data. This poses a particular challenge when bringing disparate data together for research and analysis, with many surveys in particular on isolated systems that do not interoperate easily across the architecture. It also presents risks to statistical production where reliable and timely access to data is critical.

In addition, there are multiple approaches to linking datasets within statistical research projects with analytic users favouring bespoke linkage over the standardised matching service offered through the Reference Data Management Framework (RDMF). This can lead to long timescales to prepare bespoke-linked datasets, which, among other things, can contribute to an obscuring of data discoverability and governance. However, the quality of any linkage needs to be understood and quantified for robust analysis before it is used in production.

Research into the value and linking or integration of new data sources tends to focus on specific use cases, with knowledge and understanding of the value of data held in local teams and not shared broadly across the organisation. This minimises the reuse value of data across domains.

The opportunity

Central to the evolving landscape of data sourcing is the effective integration of diverse datasets and the ability to scale any such integration. By unifying administrative records, survey responses, and data from commercial organisations, the ONS can create richer, more coherent datasets that support robust analysis. The integration process involves harmonising formats, ensuring interoperability across platforms, and establishing common standards for metadata and data quality. Data integration can facilitate more powerful analysis, reduces duplication of effort, and supports the development of insights that otherwise would not exist for users.

The move to a strategic cloud architecture for ONS data ingest, preparation and production activities provides the opportunity to improve accessibility and operability of data into analytic project environments. For example, the ONS has created a new data layer in the cloud, which enable deidentified data products to be more easily curated and accessed more quickly.

The growing complexity and volume of data sources within the ONS highlight a critical need for a robust data integration approach. To scale data integration effectively, the organisation will leverage indexed data through the RDMF. Indexing ensures that datasets can be efficiently linked, deidentified, and compared, regardless of origin or format. This not only streamlines the merging of administrative records, survey responses, and big data, but also establishes a foundation for interoperability and consistent metadata standards and support better understanding of data quality.

This approach has recently been piloted successfully to test the production of the Statistical Population Dataset (SPD), which would underpin population estimates, to standardise and automate the workflows. The project successfully demonstrated the viability of using data, indexed against the RDMF, to link datasets safely and consistently without the need for analysts to access personal identifiable data. We are now working towards implementing this approach for the 2026 production of the SPD.

Employing this framework enables the ONS to unify diverse data, reduce duplication, and support analytics, ultimately delivering statistics that align with user needs and regulatory requirements. In short, indexed data and reference frameworks are essential for scaling data integration, driving innovation, and maximising the value of the ONS's data estate.

Our commitments

We will make it easier to access our data

We will store critical data sources on our strategic cloud platforms by default, with automated pipelines and optimised end-to-end data journey to ensure timely and routine refresh. This approach is supporting access and reuse of critical data sources to improve core statistics – see Case study 2.

We will work through the new Data Governance Framework to put in place uplift plans, which enable data owners to migrate from Network Shares by making it easier to access and analyse data within our strategic platforms.

We will index our most analytically valuable data

We will use the RDMF as the default linkage methodology for all new administrative data sources, and will look to create linkable, standardised data products for our most in-demand survey data sources. We will continue working to understand and communicate the quality of the indexing so that users are clear how it can be used.

This will promote greater integration and analysis of survey, administrative and other alternative data through quick and consistent joining between datasets It will also support enhanced protection of personal data, through minimisation of data and fewer users requiring direct access to personally identifiable data.

We will share expertise and findings from integrated data projects

We will centrally record and track our key research projects into new and novel data sources and combinations of data. This will include a catalogue of the value of new data, enabling insight to be communicated across the organisation. We will develop and convene communities of expertise to support knowledge sharing around key themes, datasets, and methods which support integrating survey and non-survey data sources.

A new workstream has already been established in response to the UK Statistics Assembly called "Greater use of administrative data", which will give greater visibility of projects and methods across the Government Statistical Service.

The current landscape

Over the last three years, there has been an appetite to grow the volume of administrative and alternative data sources across the Office for National Statistics (ONS) estate, particularly to support previous objectives to design a Future Population and Migration Statistics system, and in support of the Integrated Data Programme - a cross-government data-sharing programme the ONS was asked to lead on behalf of government.  

The work to acquire data sources contributed to a growth in over 300 new assets between October 2023 and October 2025. The ONS is now consolidating the demand for new sources to ensure greater oversight and value for money.

Currently the business processes and decision making for utilising data sources is separated, with business areas submitting requirements into one of two separate workflows – for administrative and alternative data, or for survey design and commissioning. Therefore, analytical requirements for data are not always considered holistically, and with a tendency to avoid creating any instability in core production runs.

For administrative data, the availability of existing data is not always clear, which can lead to duplicative or unnecessary sourcing of data. In addition, legacy data shares are not regularly reviewed when new data shares are established, and data are stored and used across multiple, disparate systems, which do not interoperate easily.

For commercial data, recent efforts have brought greater visibility and oversight over decisions to procure data, ensuring more efficient procurements, but there has been recourse to purchasing data rather than using the ONS's legal powers to acquire data. This presents reliability risks where these data are used for core outputs, where data sourcing is not directly delivered by the ONS.

For survey data, the potential contribution of existing administrative data sources is not always understood or considered at the point of commissioning and designing changes. This creates an opportunity cost: the value of administrative data is not fully exploited to streamline survey design, support operations (for example, targeted sampling), and promote greater responses through reducing respondent burden.

The opportunity

The Survey Improvement and Enhancement Plan (SIEP) recognises the need for a programme of research looking at where and how alternative data can be used across surveys including sampling, survey design, and operations. These data (including administrative and financial transactions) may provide opportunities for us to improve the statistical quality, as well as the efficiency and effectiveness of our survey operation.

There is scope to ensure greater enterprise-level co-ordination around data sourcing, overtly bringing together knowledge and expertise to maintain an annual forward plan for any new data sources that the ONS is considering. This will in turn promote greater transparency of the ONS's approach to integrating new data to improve core statistics and build trustworthiness with the public.

Our commitments

We will agree sourcing priorities at an enterprise level

We will establish a Data Sources Board to drive improvements in the exploitation of existing data across the ONS and to provide co-ordination and agreement to sourcing new data across the Office. This Board will act as an enterprise gateway for new data sources, bringing together ONS analytical, survey and administrative data leadership to set the priorities and agree the mechanisms to sourcing new data. It will be supported by coordinated working across business areas and research teams to identify and evaluate potential new sources of data – see Case study 3.

We will drive quality, reliability, and sustainability through our sourcing approach

We will adopt key principles to guide our approach to new data sourcing – outlined at Section 10: Annex C, key principles to underpin data sourcing. This will ensure that we fully exploit the data we have, including considering use of administrative data to support survey sampling. It will also put quality, reliability and sustainability as core requirements when establishing new sources.

We will introduce a new decision-making framework and oversight to support a more consistent and sustainable approach to acquisition of commercial data sources. This will include explicit consideration of the appropriateness of using our legal powers to acquire data ahead of procurement decisions and building strategic partnerships based on value exchange to unlock the potential of novel data sources for statistics.

We will review our historic administrative data shares, to ensure the continued need and appropriate data governance arrangements are in place. We will decommission legacy data shares and pursue opportunities to automate and rationalise wherever possible.

We will be transparent about our approach to integrating new sources

The ONS already publishes on its website the data used for specific outputs, for example, the sources contributing to gross domestic product (GDP), and an overarching publication on the sources of administrative and alternative data it uses. This publication has recently been extended to include all the data used in outputs (not just those containing personal data), and those data sources the ONS is seeking to acquire, although there remain gaps, for example, where data are not accurately captured in the Information Asset Register.

Our transparency will be further improved through more engaging presentation of how our data sources contribute to core statistics. This will be progressed as part of a new "trust centre" on the ONS's website.

In addition, the new Data Governance Office will produce an annual plan, summarising the priorities for sourcing new data and integrating existing data to improve core outputs. This will be agreed and published on an annual basis. The first plan, showing priorities for 2026 to 2027 will be included in the next ONS plan for economic statistics: progress update.

This Office for National Statistics (ONS) data sourcing approach is a living framework – designed to evolve with the needs of society, advances in technology, and the changing data landscape. It sets out three strategic objectives:

• improving the quality of critical data sources

• maximising value from existing data

• delivering a transparent, coherent approach to sourcing new data, supported by new governance roles and frameworks

By embedding flexibility, transparency, and quality at every stage, and by strengthening partnerships across government and the private sector, the ONS will secure the data that matters: the foundation for trusted, high-impact national statistics.

This approach will be reviewed regularly, informed by feedback from analysts, data providers, and users, to ensure the ONS remains at the forefront of public sector data sourcing in the UK. Updates will be included as part of the quarterly ONS plan for economic statistics: progress updates, where appropriate.

Direct data collection tools

Direct data collection through surveys remains the backbone of official statistics. The Office for National Statistics (ONS) utilises a suite of tools that gather data precisely tailored to statistical requirements. These tools must evolve with technology, employing automation and ensuring a seamless user experience while allowing the ONS to collect data and address gaps in quality. The design of these tools seeks to be respondent-centred, offering a range of collection modes:

• online surveys

• face-to-face interviews

• telephone interviews

• paper forms

• digital forms (tablet, mobile, web forms, editable PDFs)

Automation, artificial intelligence (AI), and interoperability are increasingly woven into these tools, aiming for operational efficiency and low respondent burden. The ONS directly manages over 80 business and social surveys, and outsources a few other data collections to specialised partners where it enhances data quality or coverage.

Indirect administrative data from government

The ONS leverages a wide array of administrative data from across government to improve quality, coverage, and timeliness of outputs. Specifically, administrative data creates sampling frames and informs statistical design, provides variable level data which feeds statistical production, and aids quality assurance.

Administrative data supports with the production of employment, population, business, and trade statistics, for example, in admin-based population estimates. Close partnership with government departments and strong internal coordination are essential to ensure data quality and to further enable the collection of emerging data requirements, where possible. Examples of administrative data sources include HM Revenue and Customs (HMRC) Pay As You Earn (PAYE) records, Department for Work and Pensions (DWP) benefits and income, Driver and Vehicle Licensing Agency (DVLA) driver licenses, and data from the Bank of England.

Alternative data from non-government sources

To fill real-time data gaps and enable innovation, the ONS increasingly incorporates third-party and big data into its analysis and statistics. These sources – such as retail scanner data, mobile network coverage, satellite imagery, credit card spend data, and mortgage data – offer new data and aid with new perspectives on the economy and society.

Strong engagement with private sector partners, trade bodies, regulators, and non-departmental public bodies is critical for acquiring and integrating these data sources. This benefits both the ONS and wider users of ONS data, including to support effective decision-making and insights into wider areas of the economy and society, such as the housing market, tax and spending, debt, financial forecasting and much more.

In late 2023, the Office for National Statistics (ONS) undertook an exercise to identify its most important microdata assets that are either shared directly with other public bodies or accessed by accredited researchers via trusted research environments. This was in response to a commission by the Central Digital and Data Office in Cabinet Office, with the expectation that information about these assets would be published via the Data Marketplace.

This process considered the quantitative demand for assets from external users, alongside the relative importance of these usages. The Data Strategy team in the ONS has subsequently worked with information asset owners (IAOs) to provision and maintain metadata for these essential shared data assets (ESDAs) to support discoverability and useability.

The initial 20 ESDAs will be reviewed and updated iteratively:

• Inter Departmental Business Register (IDBR)

• Labour Force Survey (LFS)

• Annual Survey of Hours and Earnings (ASHE)

• Annual Population Survey (APS)

• Annual Business Survey (ABS)

• Opinions and Lifestyle Survey

• Census 2021

• Business Enterprise Research and Development (BERD)

• Business Register Employment Survey (BRES)

• International Trade in Services (ITIS)

• PRODCOM

• Annual Purchases Survey

• Census 2011 - England and Wales

• Foreign Direct Investment

• Household Finance Surveys (HFS)

• Business Insights and Conditions Survey (BICS)

• Mortality

• Management and Expectations Survey

• Consumer and Retail Prices Index

• Births

1. Prioritising critical data sources: focusing effort on the most analytically valuable and in-demand data sources for Office for National Statistics (ONS) needs.

2. Building quality into every step: addressing quality needs at the earliest stage and ensuring data quality is embedded in all activity.

3. Improving data reliability: strengthening stability, consistency, and resilience of data supply, especially for critical sources.

4. Sustainability:considering financial sustainability and using our legal framework to acquire data to full potential.

5. Maximising data reuse: enabling cross-ONS reuse of existing data, reducing duplication and burden, and supporting integrated statistical models.

6. Timeliness and granularity: pursuing more real-time, granular, and inclusive outputs by adapting to new sources and methods.

7. Meeting user needs and reducing burden: decreasing the effort required from data providers and survey respondents by using smart and respondent-tailored tools.

8. Mitigating single-source risk: diversifying data sources to avoid over-reliance on any single channel or supplier.