Consumer prices development plan: updated July 2023

The Office for National Statistics (ONS) has a programme of research and development aimed at improving and maintaining its range of consumer price inflation statistics. The programme will ensure that the statistics continue to meet user needs, make use of new and innovative methods and data sources, and follow international best practice.

Our Measuring changing prices and costs for consumers and households, proposed updates: March 2020 article, provides more detail on the "use cases" for each of our main inflation measures, including:

• Consumer Prices Index including owner occupiers' housing costs (CPIH)
• Consumer Prices Index (CPI)
• Household Costs Indices (HCIs)
• Retail Prices Index (RPI)

The UK statistics strategy, Statistics for the public good (2020 to 2025), sets the collective mission for the official statistics system as, "High quality data and analysis to inform the UK, improve lives and build the future". It is based on four pillars: radical, ambitious, inclusive, and sustainable.

Our development plan reflects how our consumer prices development work contributes to the strategy. We contribute by efficiently producing high quality and relevant price statistics to meet users' needs, while keeping pace with evolving methods, sources, and digital processes. 

Items on the development plan are prioritised through discussion with our Advisory Panels on Consumer Prices (APCP).

Our current scope of work is focused within the high priority items, and therefore we do not expect to make progress on the medium and low priority work in the near future. Medium priority items will be pursued as resource allows without impacting progress on high priority items. Low priority items will only be taken forward where resource is available and will not prevent high or medium priority workstreams from progressing.

The work programme for consumer price statistics has been updated to reflect the views of the Advisory Panels on Consumer Prices, following discussion with both the Technical and Stakeholder Panels, and, indirectly, feedback from wider stakeholders. For example, wider feedback was gained through the Office for National Statistics (ONS) user engagement surveys and events like our Understanding the cost of living through statistics event.

Some aspects of the current work programme reflect the outcome of the Johnson Review 2015 (UK Consumer Price Statistics: A Review - UK Statistics Authority), led by Paul Johnson, Director of the Institute for Fiscal Studies. They also reflect the priority placed on the use of alternative data sources, as found in the Bean Review (Independent review of UK economic statistics: final report - GOV.UK), led by Professor Sir Charles Bean of the London School of Economics.

Recent work, since our Consumer prices development plan: updated July 2022 article, includes:

• September 2022 - published our pilot work on Boosting the Northern Ireland price sample for the Consumer Prices Index methodology

• October 2022 - published experimental analysis looking at the lowest cost grocery items in our Tracking the price of the lowest-cost grocery items, UK, experimental analysis: April 2021 to September 2022 article

• November 2022 - published research into new index number and outlier detection methods for alternative data sources in our Research and developments in the transformation of UK consumer price statistics: November 2022 article

• February 2023 - published our Impact analysis on transformation of UK consumer price statistics: rail fares, February 2023 article

• March 2023 - incorporated transaction level rail fares data into headline measures of inflation, as shown in our Impact analysis on transformation of UK consumer price statistics: rail fares, February 2023

• May 2023 - published the shopping prices comparison tool, a new interactive tool to understand the price changes of different items in a consumer price basket

• May 2023 - published our Boosting the Northern Ireland price sample for the Consumer Prices Index, including experimental regional weighting methodology

• Historical changes to consumer price statistics can be found in Section 15 of our Consumer Prices Indices Technical Manual, 2019 methodology

High priority items 

High priority items are the main element of our development programme and, if necessary, will be prioritised over the delivery of medium and low priority items.

Transformation of consumer price statistics 

We are currently working through a comprehensive transformation programme for consumer price statistics, to modernise their measurement and make better use of data and methods that become available to us. This is currently our highest priority, other than producing monthly outputs, and will deliver the largest improvement to consumer price statistics in a generation.

At a high level, this involves obtaining robust sources of alternative data and developing an innovative and scalable end-to-end production system, on a cloud platform, to process both new and traditional data sources. This development modernises the measurement of price indices constructed using traditional data sources, while enabling processing and integration of new data sources for specific categories.

This programme also involves methodological research to effectively classify, validate and construct high quality price indices from new data sources. There will be research into how existing measurements will need to adapt to enable integration of new data sources. We will also consider how these new processes can be effectively embedded into the existing monthly business process. These new data sources will be integrated with traditionally collected data using new systems and methods to improve the accuracy, efficacy, and representativity of consumer price inflation statistics. 

We are investigating a range of data sources for different item categories. For more information, see our Transformation of consumer price statistics: July 2023 article, which includes our roadmap, that sets out our plans to continue to incorporate alternative data sources into our headline measures of consumer price statistics.

Since 2020, we have worked on developing the methods and systems for alternative data sources. Our Research and developments in the transformation of UK consumer price statistics articles series, released biannually, provides an update to users on our work.

The redevelopment of private rental prices statistics are anticipated to be used to measure private rental prices in consumer price statistics and owner-occupiers' housing costs in the Consumer Prices Index including owner occupiers' housing costs (CPIH) in the future. The current phase of the project involves the publication of aggregate research indices, using new data and methods, as well as a range of impact analyses. For more information on the redevelopment of private rental prices statistics, see The redevelopment of private rental prices statistics, intended methodology: March 2022.

We will continue to gather feedback from users on our communication and publications approach to ensure we are best meeting stakeholder needs. In early 2023 we published our Impact analysis on transformation of UK consumer price statistics: rail fares, February 2023 article. We are currently working through our publication schedule for second-hand cars, private rents and groceries.

Continuous improvement in sources and coverage

Alongside our current transformation project, we continually review the quality and relevance of our consumer price statistics. This work informs our priorities for future development, either as part of ongoing maintenance of our statistics or potential future transformation priorities.   

An example of this wider improvement work is our collaboration with the Office of Gas and Electricity Markets (Ofgem). The work aims to expand our coverage of gas and electricity tariffs in our consumer price statistics to include more fixed rate tariffs. Subject to further feedback from our Advisory Panels for Consumer Prices (APCP), we are working towards including this improved source in our March 2024 basket update. 

Developing the Household Cost Indices (HCIs) 

Work to develop the HCIs - measures of price change as experienced by different household groups - was first suggested in the Johnson Review 2015 (UK Consumer Price Statistics: A Review - UK Statistics Authority), before being proposed as the Household Inflation Index, by Astin and Leyland. Preliminary experimental HCIs were first published by the Office for National Statistics (ONS) in 2017, the Household Costs Indices, UK: fourth preliminary estimates, 2005 to 2021 bulletin was published in May 2022.

We are committed to developing and delivering the HCIs as a quarterly measure later this year, as shown in our Statement on the future of the Household Costs Indices: June 2022. With the current rise in the cost of living and an increased stakeholder need for more timely estimates, we will develop a sustainable system to produce the existing HCIs on a quarterly basis.

Timely and innovative analysis

With recent rises in inflation and the cost of living, we have prioritised work to provide additional insights for users using new, innovative analysis and the development of our current measures, such as the HCIs.

Recent new insights include innovative tools, such as the shopping prices comparison tool and the personal inflation calculator. There have been wider stakeholder engagement events, such as an ONS-led understanding the cost of living through statistics event on 25 October 2022.

We have also prioritised new timely analysis such as:

• experimental analysis looking at the lowest-cost grocery items analysis, as shown in our Tracking the price of the lowest-cost grocery items, UK, experimental analysis: April 2021 to September 2022 article

• food and energy analysis, as shown in our Food and energy price inflation, UK: 2023 article

• changes in private rental sector behaviour in England experimental analysis, as shown in our Changes in private rental sector behaviour, England: February 2022 to February 2023 article

Classification of individual consumption by purpose (COICOP) 2018 implementation

COICOP is the classification structure that underpins CPIH and the Consumer Prices Index (CPI). It is also used to construct Household Final Consumption Expenditure (HHFCE) estimates as part of the System of National Accounts. These data are subsequently used to weight the CPIH and CPI basket of goods and services. Revisions to the COICOP structure were finalised by the United Nations (UN) in 2018, as shown in the UN's COICOP Revision. Work will therefore be required to ensure we can restructure CPIH and CPI to account for these changes, to remain in line with international best practice and standards.

Medium priority items 

Medium priority items form an important part of the work programme. However, with our current focus on high priority items, we are unable to progress medium priority items until high priority items have progressed.

Developing measures of accuracy for CPIH 

As a result of the complex survey design, calculating standard errors for CPIH and, the growth in CPIH, is challenging. With two dimensions to the sampling: selection of items and selection of outlets, both making the estimation of sampling errors difficult.

An article on the effect of variance in the weights of CPIH was published in Autumn 2017, and work on the effect of variance in the prices was reviewed by the APCP in 2018. As a result of this work, an academic review on producing standard errors for consumer price indices was published, Estimating Sampling Errors in Consumer Price Indices (Wiley Online Library). We plan to publish further work, including:

• a publication of interim estimates of components of the CPI sampling variance based on CPIH data 
• an academic paper presenting estimates of components of the CPI sampling variance based on CPIH data 
• extending variance estimation to new forms of data to be included in CPIH and CPI (rail fares transaction data, second-hand cars web-provided data, grocery scanner data, rental microdata, and web-scraped data)

Improvements to owner occupier housing costs methodology 

Alongside the work to redevelop our private rental prices statistics (which is a high priority as part of the consumer prices transformation workstream), there are other ways in which the owner occupiers' housing costs (OOH) component of CPIH could be improved. For more information on this, see our Private rental prices development plan, UK: updated February 2022 article.

The Johnson Review 2015 (UK Consumer Price Statistics: A Review - UK Statistics Authority) suggests a flow measure may be worth considering; that is, only new lets. We investigated the feasibility of measuring the flow of rents and concluded that we do not currently have data sources available to us and will review as new data sources become available.

Northern Ireland rent prices are currently collected as part of the wider CPIH price collection, but we will look investigate the potential of using data from the Northern Ireland Housing Executive (NIHE) to improve the sample.

Further quality assurance of CPIH 

As part of the work to seek re-accreditation for CPIH as a National Statistic, we developed a Quality assurance of administrative data (QAAD) used in consumer price inflation statistics methodology. The QAAD highlighted several areas where the quality assurance for some data sources could be improved. We continue to seek the required assurance for these sources and will aim to update the QAAD biennially. 

Barcode pilot

Since June 2021, we have run a pilot collection of Global Trade Item Numbers (GTINs or barcodes) when collecting prices from physical outlets. We aim to use this to match products and prices more effectively between our alternative and traditional data sources. This will enable improved quality assurance procedures and the potential to directly substitute traditional data with alternative data.

Review of quality adjustment and monitoring of quality change 

In response to the Johnson Review 2015 (UK Consumer Price Statistics: A Review - UK Statistics Authority), Bean Review (Independent review of UK economic statistics: final report - GOV.UK), and stakeholder engagement, we aim to continue to develop tools for monitoring quality adjustment methods used in consumer price inflation statistics and provide more detail on how quality adjustment is monitored. 

Low priority items 

The delivery of low priority items is currently paused to ensure the delivery of high priority items. 

Regional consumer price indices 

The regular collection of prices for consumer price inflation statistics is optimised for measuring inflation at the UK level. This means that the number of locations visited per region may be insufficient for regional estimation. A previous analysis was explored with the University of Southampton in November 2017 and February 2019.

We have recently piloted producing experimental subnational consumer prices for Northern Ireland. In September 2022 we published our Boosting the Northern Ireland price sample for the Consumer Prices Index methodology and then in May 2023 we published our Boosting the Northern Ireland price sample for the Consumer Prices Index, including experimental regional weighting methodology. This looked at the feasibility of improving the granularity of the statistics by boosting the CPIH sample in this region. Further information is detailed in a case study in the Government Statistical Service (GSS) subnational data strategy.

While regional price indices are an important interest for some stakeholders, we have no current plans to do further work in this area.

Improvements to elementary aggregate indices 

In 2010, the ONS made several changes to the methodology used to collect clothing prices. These changes meant that the gap between Retail Prices Index (RPI) and CPI, which use different formulae at the lowest level of aggregation, widened. For more information, see our CPI and RPI methodology (ONS National Archives).

We will consider recommendations from the Johnson Review 2015 (UK Consumer Price Statistics: A Review - UK Statistics Authority) to review and publish the criteria for formula selection at the lowest level of aggregation.

Improvements to owner occupiers' housing costs net acquisitions (OOH(NA)) 

We previously produced an experimental net acquisitions index for Eurostat, as part of a pilot to incorporate OOH costs into the Harmonised Index of Consumer Prices (HICP). The index was included in a comparison article comparing different approaches of measuring OOH. We will explore data sources that could be used to improve this experimental index.

Inclusion of financial intermediation services indirectly measured (FISIM) in CPIH 

FISIM are included in the National Accounts measure of household final consumption expenditure (HHFCE) but are not currently included in consumer price indices. The exclusion of FISIM is one of the biggest differences between the two. As CPIH is not bound by the same legislation, we will consider the suitability of including FISIM as a proxy for the service charge that households pay to banks.

Discounting pilot

We have been collecting data on discounts alongside our consumer prices for several years. Discounts are not currently included in consumer price statistics as there is little information as to how many consumers take-up the available offers. In the future, we will look to understand the take-up rates of discounts, using data from our scanner retailers.

We will continue to keep users updated with the progress of our consumer prices development plan and publish an update of this article annually.