Consumer prices development plan: updated August 2024

The Office for National Statistics (ONS) has a programme of research and development aimed at improving and maintaining its consumer price inflation statistics. The development plan will ensure that the statistics continue to meet user needs, make use of new and innovative methods and data sources, and follow international best practice.

The delivery of quality prices statistics is a priority for the ONS and forms a recognised part of our core outputs. As we move towards a new Spending Review period, we will continue to prioritise our core economic statistics that are important for informing decisions related to the UK economy and the public.

We have now implemented alternative, big data changes into headline statistics for several categories, starting with rail fares (launched in March 2023), followed by private rents (for Great Britain) and second-hand cars (launched in March 2024).

Our next high-priority developments are for grocery scanner data and Northern Ireland private rents, which we will implement in March 2025. We are also updating and modernising our statistical production system to process data.

Our Measuring changing prices and costs for consumers and households, proposed updates: March 2020 article provides more detail on the "use cases" for each of our main inflation measures, including:

• Consumer Prices Index including owner occupiers' housing costs (CPIH)
• Consumer Prices Index (CPI)
• Household Costs Indices (HCIs)
• Retail Prices Index (RPI)

As part of the work to seek re-accreditation for CPIH as accredited official statistics, we developed a Quality assurance of administrative data (QAAD) used in consumer price inflation statistics methodology. The QAAD highlights several areas where the quality assurance for some data sources could be improved, and currently provides information on how we will maintain and improve the quality of the data sources we use. We continue to seek the required assurance for these sources and will aim to update the QAAD as required. The last update was in March 2024. 

The UK Statistics Authority strategy, Statistics for the public good (2020 to 2025), sets the collective mission for the official statistics system as: "High quality data and analysis to inform the UK, improve lives and build the future". It is based on four pillars: radical, ambitious, inclusive, and sustainable.

Our development plan reflects how our consumer prices development work contributes to the strategy. We contribute by efficiently producing high-quality and relevant price statistics to meet user needs, while keeping pace with evolving methods, sources, and digital processes.

Items on the development plan are prioritised through discussion with our Advisory Panels on Consumer Prices (APCP) and other stakeholders.

Our current work is focused on high-priority items. In general, we will only progress medium- or low-priority items if that can be done with no impact on the progress of the high-priority items.

The development plan for consumer price statistics has been updated to reflect the views of the Advisory Panels on Consumer Prices (APCP), following discussion with both the Technical and Stakeholder Panels, and feedback from other stakeholders.

Some aspects of the current development plan reflect the outcome of the UK Statistics Authority's UK Consumer Price Statistics Review (Johnson Review) led by Paul Johnson, Director of the Institute for Fiscal Studies. The development plan also reflects the emphasis placed on the use of alternative data sources in the Independent review of UK economic statistics: final report (GOV.UK) (Bean review) led by Professor Sir Charles Bean of the London School of Economics.

Alongside this plan, we have also published our Producer prices development plan article in response to the Office for Statistics Regulation (OSR) Spotlight on Quality: Producer Price Indices (PPIs). Reviewing and updating the survey samples and replacing the legacy Ingres-based system currently used to produce PPIs with a more robust, flexible and sustainable statistical production system are a high priority.

Since our Consumer prices development plan: updated July 2023 article we have:

• published an indicative impact analysis of the planned improvements to private rental and second-hand car measurement in consumer price statistics (December 2023)

• published an indicative impact analysis of the redevelopment of private rental price statistics (December 2023)

• begun quarterly publication of our Household Costs Indices bulletins (December 2023)

• incorporated improved private rental prices and second-hand car data into consumer price statistics, as shown in our Consumer price inflation, UK: February 2024 bulletin (March 2024)

High-priority items

High-priority items are the main element of our development plan and will usually be prioritised over the delivery of medium and low-priority items.

Transformation of consumer price statistics 

We are working through a comprehensive transformation programme for consumer price statistics, to modernise their measurement and make better use of data and methods that become available to us. This is currently our highest priority, other than producing monthly outputs, and will deliver one of the largest improvements to consumer price statistics in a generation.

At a high level, this involves obtaining appropriate sources of alternative data and developing an innovative and scalable end-to-end production system, on a cloud platform, to process both new and traditional data sources. This development modernises the measurement of price indices constructed using traditional data sources, while enabling the processing and integration of new data sources for specific categories.

This programme also involves methodological research to classify, validate and construct high-quality price indices from new data sources. We will conduct research into how existing measurements will need to adapt to enable the integration of new data sources. We will also consider how these new processes can be embedded effectively into the existing monthly business process. These new data sources will be integrated with traditionally collected data using new systems and methods to improve the accuracy, effectiveness, and representativeness of consumer price inflation statistics.

We are investigating a range of data sources for different item categories. For more information, see our Transformation of consumer price statistics: August 2024 article. This article includes our roadmap, which sets out our plans to continue to incorporate alternative data sources into our headline measures of consumer price statistics.

Since 2020, we have developed our methods and systems for working with alternative data sources. Our Research and developments in the transformation of UK consumer price statistics articles series, released biannually, provides an update to users on our work.

We have also made progress towards the implementation of grocery scanner data in consumer price statistics in March 2025. The current phase of the project involves the publication of aggregate research indices, using new data and methods, as well as a range of impact analyses.

On a monthly basis, we currently publish much of the price microdata used to calculate consumer price inflation statistics (specifically the Consumer price inflation item indices and price quotes). With the introduction of scanner data, the breadth of microdata that we currently publish will be reduced. This is partly because of the much greater size and the different structure of this new data source. However, our data sharing agreements with suppliers also prevent us from publishing raw scanner datasets. We are looking for feedback on how the price quote microdata and item indices are currently used. Further information is available on our consultation webpage. This will inform our future data publication strategy for scanner data and whether any new aggregate outputs need to be produced.

Following the redevelopment of private rental price statistics in March 2024 for England, Wales and Scotland, work has continued towards the improvement of Northern Ireland private rental price statistics. Currently, Verian (previously Kantar) data are used to measure Northern Ireland rents in consumer price statistics. We will investigate the potential of using rents data from the Northern Ireland Housing Executive (NIHE) to improve the sample and methodology used to derive Northern Ireland rents in consumer price statistics.

We will continue to gather feedback from users on our communication and publications approach, and we are currently working through our publication schedule for grocery scanner data and Northern Ireland private rental price statistics.

Continuous improvement in sources and coverage

Alongside our current transformation project, we continually review the quality and relevance of our consumer price statistics. This work informs our priorities for future development and our ongoing maintenance of our statistics.

To enable the introduction of grocery scanner data in March 2025, our statistical production system is being updated and modernised. More information is available in our Transformation of consumer price statistics: August 2024 article. We are also reviewing our current methods and will be making changes in March 2025. In particular, we will be reducing the lag at which missing prices are imputed. We will also introduce flexibility into the system to align imputations with the defined index method for each "use case". More information on imputation processes is available in Section 9 of our Consumer prices indices technical manual. More information on the consumer prices use cases is available in our Measuring changing prices and costs for consumers and households article.

Furthermore, we will change the stratification of some items to allow for the introduction of consumption segments. We will also review our methods for the calculation of retailer weights. More information is available in our Introducing alternative data into consumer price statistics: aggregation and weights article. We will publish these changes in a series of analysis articles towards the end of 2024.

Another example of this wider improvement work is our review of how we sample gas and electricity tariffs. We aim to expand our coverage in our consumer price statistics to include more fixed rate tariffs. Subject to further feedback from our Advisory Panels for Consumer Prices (APCP), we are working towards including this in our March 2025 basket update. 

Medium-priority items

Medium-priority items form an important part of the development plan. However, with our current focus on high-priority items, we are only able to progress medium-priority items if the work does not impact on the progress of the high-priority items.

Classification of individual consumption by purpose (COICOP) 2018 implementation

COICOP is the classification structure that underpins the Consumer Prices Index including owner occupiers' housing costs (CPIH) and the Consumer Prices Index (CPI). It is also used to construct Household Final Consumption Expenditure (HHFCE) estimates as part of the System of National Accounts. These data are subsequently used to weight the CPIH and CPI basket of goods and services. Revisions to the COICOP structure were finalised by the United Nations (UN) in 2018, as shown in the UN's COICOP Revision. Work will therefore be required to restructure CPIH and CPI to account for these changes, ensure they are comprehensive and consistent across other economic statistics, and that they remain in line with international best practice and standards.

Seasonally adjusted CPI

The ONS have commissioned the National Institute of Economic and Social Research (NIESR) to carry out a feasibility study on the most appropriate methodology to use in the production of seasonally adjusted consumer prices inflation. We do not currently publish a seasonally adjusted CPI but see value in investigating the possibility. The project will report back by January 2025, with the intention of providing the ONS with an appropriate approach to produce a monthly, seasonally adjusted set of indices that can be published as supplementary tables alongside the headline statistics.

Developing the Household Costs Indices

The Household Costs Indices (HCIs) measures price change as experienced by different household groups. It was first suggested in the UK Statistics Authority's UK Consumer Price Statistics Review (Johnson Review) in 2015. It was then proposed as a Household Inflation Index by John Astin and Jill Leyland, in their paper for the Royal Statistical Society (RSS) Compiling a consumer price index with public credibility (PDF, 947KB). Preliminary HCIs were first published by the ONS in 2017.

We now publish our Household Costs Indices for UK household groups bulletin quarterly. HCIs are currently classed as official statistics in development so future work will focus on seeking accredited official statistics status.

Developing measures of accuracy for consumer price statistics

As a result of the complex survey design, calculating standard errors for the level and growth of CPIH is challenging. There are two dimensions to the sampling (the selection of items and the selection of outlets); both make the estimation of sampling errors difficult.

Our survey methodology bulletin on the The effect of variance in the weights on the CPI and RPI (PDF, 2.5MB) was published in Autumn 2017. Our work on the effect of variance in the prices was also reviewed by the Advisory Panels for Consumer Prices (APCP) in 2018. As a result of this work, we worked with the University of Southampton to publish an academic review, Estimating Sampling Errors in Consumer Price Indices (Wiley Online Library). We plan to publish further work with the university, including:

• interim estimates of components of the CPI sampling variance, based on CPIH data

• an academic paper presenting estimates of components of the CPI sampling variance, based on CPIH data

We also plan to work with the university to extend variance estimation to new forms of data to be included in CPIH and CPI (rail fares transaction data, second-hand cars web-provided data, grocery scanner data, rental microdata, and web-scraped data).

Review of quality adjustment and monitoring of quality change

In response to the Johnson Review, Bean Review, and stakeholder engagement, we aim to continue to develop tools for monitoring quality adjustment methods used in consumer price inflation statistics and provide more detail on how quality adjustment is monitored. 

Timely and innovative analysis

During 2021, 2022 and 2023, with rises in inflation and the cost of living, we prioritised work to provide additional insights for users; for example, through our shopping prices comparison tool and personal inflation calculator. We also led stakeholder engagement events, such as the Understanding the cost of living through statistics event on 25 October 2022.

With less current need for additional timely analysis, the priority of these analyses has been lowered to a medium.

Low-priority items

The delivery of low-priority items is currently paused to ensure the delivery of high-priority items. 

Regional consumer price indices

The regular collection of prices for consumer price inflation statistics is optimised for measuring inflation at the UK level. This means that the number of locations visited per region may be insufficient for regional estimation. A previous analysis was explored with the University of Southampton in November 2017 and February 2019.

We have recently piloted producing experimental subnational consumer prices for Northern Ireland (NI). This work has been funded by the Department for the Economy, the Consumer Council for Northern Ireland, and the Northern Ireland Statistics and Research Agency (NISRA).

In September 2022 we published our Boosting the Northern Ireland price sample for the Consumer Prices Index (CPI) methodology; in May 2023 we published our Boosting the Northern Ireland price sample for CPI, including experimental regional weighting methodology. This explored the feasibility of improving the granularity of the statistics by boosting the Consumer Prices Index including owner occupiers' housing costs (CPIH) sample in this region. Further information is detailed in a case study in the Government Statistical Service (GSS) subnational data strategy.

Working with stakeholders in NI, we are continuing to assess the feasibility of producing an experimental NI CPI.

Barcode pilot

Since June 2021, we have run a pilot collection of Global Trade Item Numbers (GTINs or barcodes) when collecting prices from physical outlets. We aim to use this to match products and prices more effectively between our alternative and traditional data sources. This will enable improved quality assurance procedures and the potential to directly substitute traditional data with alternative data.

Improvements to elementary aggregate indices

In 2010, the ONS made several changes to the methodology used to collect clothing prices. These changes meant that the gap between the Retail Prices Index (RPI) and CPI, which use different formulae at the lowest level of aggregation, widened. For more information, see our CPI and RPI methodology (National Archives).

We will consider recommendations from the 2015 Johnson Review to review and publish the criteria for formula selection at the lowest level of aggregation.

Discounting pilot

We have been collecting data on discounts alongside our consumer prices for several years. "Buy one get one free" and similar discounts are not currently included in consumer price statistics, as there is little information about how many consumers take up the available offers. In the future, we will look to understand the take-up rates of these discounts, using data from our scanner retailers.

Improvements to owner occupier housing costs methodology

Alongside recently completed work to redevelop our private rental prices statistics, there are other ways in which the owner occupiers' housing costs (OOH) component of CPIH could be improved. For example, improvements to the Northern Ireland private rental series, which are detailed in our high-priority items.

The 2015 Johnson Review suggests a flow measure for new lets may be worth considering. We investigated the feasibility of measuring the flow of rents and concluded that we do not currently have the appropriate data sources available to us. We will review this as new data sources become available.

Editions of this article are published annually.

If you have any feedback or suggestions on the work we are doing with consumer price statistics, please contact us at cpi@ons.gov.uk.