Consumer Prices Index including owner occupiers' housing costs (CPIH) historical series: 1988 to 2004

• We have modelled a historical series from 1988 to 2004 for the Consumer Prices Index including owner occupiers’ housing costs (CPIH).

• CPIH peaked at 9.2% between September and December 1990 following sustained economic growth and rapid house price growth in the late 1980s.

• The housing, water, gas, electricity and other fuels category consistently contributed the most to the headline rate, with the largest contribution amounting to 74.6% of headline CPIH in March 2000.

• There were large increases in the average contribution of education, which more than doubled from an average of 0.06 percentage points in the 1989 to 2004 period, to an average of 0.13 percentage points in the 2005 to 2017 period.

Consumer price inflation is the rate at which the prices of goods and services bought by households for the purpose of consumption rise or fall. The Consumer Prices Index including owner occupiers’ housing costs (CPIH) is our lead measure of consumer price inflation. It has the same coverage as the Consumer Prices Index (CPI, the lead measure of inflation up until March 2017), but with the addition of Council Tax and owner occupiers’ housing costs (OOH).

OOH costs are the costs associated with owning, maintaining and living in one’s own home and is distinct from the purchase of a house, which can be viewed as the purchase of an asset. CPIH uses the “rental equivalence” approach, which determines how much rent would be paid for an equivalent property in the private sector, to estimate owner occupiers’ housing costs. It aims to capture the ongoing consumption of housing services.

CPIH has previously been calculated back to 2005, which is the earliest that suitable data sources are available for constructing a rental equivalence index, while CPI component series are available back to 1996. This article reports on work done to produce a back-history for the rental equivalence price index and for CPIH back to 1988.

As well as a longer CPIH time series, users will now be able to access more detailed CPI historical estimates for the first time. The data are being published at the class level (the fourth level of the international Classification of Individual Consumption by Purpose, or COICOP). This is more detailed than previous CPI historical estimates, which were only available at the division level (the second COICOP level) between 1988 and 1996.

In developing the series, we identified an error in the calculation of the modelled CPI historical estimates. This does not affect the CPI National Statistic series published from 1997. The affected part of the series is between 1988 and 1996, which was modelled later, after the introduction of CPI in 1997. As the series was modelled after the event, the data have never been used for any policy or uprating purposes. We will be revising the series to correct this error. The maximum absolute revision to the all-items 12-month growth rate is 0.2 percentage points in two periods. Of the 96 months affected, there are 10 revisions in total at this level. The average absolute revision is 0.01 percentage points. For more information, and for lower-level revisions, please refer to Annex A.

The statistics in this release are classified as official statistics. They form part of a longer series for the CPIH, of which the latter part, from 2005 onwards, constitutes a National Statistic. This modelled series covers the earlier period from 1988 to 2005 and has been constructed using historical data. As an official statistic, the modelled CPIH historical series meets the standards set out in the Code of Practice for Statistics. However, there are limitations with constructing a historical series and, as such, users should treat these estimates with some caution. For more detail, please refer to Annex B. Indices will be published to three decimal places in the consumer price inflation tables. For more information on how CPIH is constructed from 2005 onwards, see the Consumer Price Indices Technical Manual and the CPIH compendium, which provides specific details on the rationale behind the OOH measure and how it is constructed in practice.

Background

The UK’s Harmonised Index of Consumer Prices (HICP), now known as the Consumer Prices Index (CPI), was first published in 1997. The index officially starts in 1996 but when it was first published, estimates were also produced for 1995. The following year, in 1998, earlier data for 1988 to 1994 were published broken down into the 12 divisions of the Classification of Individual Consumption by Purpose (COICOP)¹. The more detailed series, at COICOP group level (for example, food) and COICOP class level (for example, fruit), that were calculated at that time are now being made available for the first time.

The Consumer Prices Index including owner occupiers’ housing costs (CPIH) was introduced in 2013 following extensive development work with the Consumer Prices Advisory Committee between 2009 and 2013. It introduced a “rental equivalence” measure of owner occupiers’ housing costs (OOH) into CPI. A Council Tax index (rates for Northern Ireland) was subsequently added in 2017 and CPIH was revised to incorporate this additional element across the entire time series. The series began in 2005, which is the earliest point for which we have a robust source of private rental data.

The rental equivalence index that is currently used in CPIH from 2005 is constructed using administrative data on private rents collected by the Valuation Office Agency (VOA) in England, for the purposes of administering Housing Benefit and Universal Credit functions. The VOA data provide around 500,000 private rental prices a year, and the dataset is adjusted to reflect the composition of the owner-occupied housing market. We are also given private rental prices by Rent Officers Wales in Wales and by Rent Service Scotland in Scotland. Northern Ireland rent prices are currently collected as part of the wider CPIH price collection. For more information, please see the CPIH compendium.

Calculation methods

This section describes how the CPIH estimates for 1988 to 2004 were produced.

Price indices for rental equivalence and Council Tax or rates

Estimates of CPIH back to 1988 have been produced by taking the existing estimates for the CPI for 1988 to 1995 and adding in indices for rates or Council Tax and rental equivalence.

The price index for rates or Council Tax index is that used in the Retail Prices Index (RPI). The index is calculated by weighting the percentage increase in each local authority’s Council Tax by their share of total expenditure on Council Tax. This means that issues around the formula used elsewhere in the RPI where weighting information is not available do not arise.

The rental equivalence index is constructed from the historic private rental indices used in CPI (January 1996 to December 2004) and RPI (January 1988 to December 1995). The CPI and RPI private rentals indices were found to correlate closely with the rental equivalence index after 2005, leading it by four months. This can be seen in Figure 1. Various modelling approaches were tried to produce a backcast index (essentially, forecasting backwards in time), but the models were found to have very wide confidence intervals. This suggested that there was little to distinguish between different approaches in terms of quality (more information is available in Annex C).

It was therefore decided to construct the CPIH historical series from existing historical data sources, accepting the limitations involved in this approach. Where available, CPI indices are preferred; private rental indices are available back to 1996. As part of the CPI (and CPIH) modelled historical series, rental indices are available in earlier periods; however, these are at a higher level than the private rental indices and therefore also include social housing. Rental prices for social housing are inappropriate for a measure of owner occupiers’ housing costs (OOH) as rental prices used should be unregulated. Analysis suggests that the upward bias that would be introduced by using social rents is greater than the formula bias from use of the RPI private rents series². In fact, the formula effect for private rental indices is demonstrably small. The RPI private rents series is therefore preferred prior to 1996.

Users should be aware, however, that there are limitations to using historic rental data in this way. The indices are based on locally collected price quotes, rather than the large administrative sources used in CPIH after 2005. The sample is optimised at the national level to produce robust estimates of price change for the private rental market. However, the sample is not large enough to be adjusted to reflect the composition of the owner-occupied housing market. This means that the historic imputed rents series is based on the assumption that the composition of the private rental and owner-occupied housing markets is the same (or at least is adequately reflected in the rental data). This assumption is unlikely to be justified in practice.

In the preceding paragraphs, we discussed different biases that could arise from using the various available data sources. In Table 1 we have attempted to quantify these by comparing the respective price indices with the live OOH series between January 2006 and December 2012. The bias is the average deviation from the live OOH series. Clearly, there are some limitations with this approach, such as:

• we would not necessarily expect these relationships to hold in earlier periods, and we might expect this to be less reliable the further back in time we go

• the results may be unduly affected by the unusual behaviour in the recession period, which may not hold in normal conditions

• CPIH has been used as the target; however, CPIH estimates, as with any measure of inflation or statistic, will also be subject to unquantifiable errors

• it is not possible to disentangle microdata differences from other biases; in other words, there will inevitably be differences due to using the locally collected data rather than the VOA data that are not isolated here

Nevertheless, these estimates do provide some indication of the relative suitability of different sources, and it is useful to have a framework within which to make decisions.

The data in Table 1 suggest that the CPI and RPI private rents series should be used in preference to the higher-level actual rents series. CPI and RPI private rents have biases of negative 0.24 and negative 0.58 respectively, whereas the equivalent biases for the CPI and RPI actual rental series are bigger, at 1.10 and 1.16 respectively. Graphical comparisons were also used to support this analysis, and agreed with the general findings.

The RPI private rentals housing index is available from January 1988. So, given data availability and the four-month lag, the modelled OOH index is available from May 1988. For the period January to April 1988, the rental equivalence deflator³ from the 2014 UK National Accounts⁴ household final consumption expenditure was used, with straight line interpolation between the quarterly data points to get a monthly series. The 12-month rate of change of the rental equivalence index can be seen in Figure 2.

CPIH weights

Expenditure weights for Council Tax and rental equivalence have been derived from the UK National Accounts. This is the same source that is used for the National Statistic part of the series from 2005 onwards. These weights have been calculated as a proportion of total household final expenditure (excluding narcotics, gambling, prostitution, financial intermediation services indirectly measured (FISIM) and life insurance, which are not included in the CPI). The weights of these two items in year t were calculated by price-updating expenditure in year t-2 to December in year t-1 in line with what was done for 2005 to 2017 (for example, the index for 2004 is based on the UK National Accounts expenditure for 2002⁵).

The weights in parts per thousand for Council Tax and rental equivalence are shown in Table 2, with the weights used for 2005 shown for comparative purposes.

The COICOP class weights for CPIH during 1988 to 2004 have been calculated by taking the existing CPI weights and rescaling them to their total excluding Council Tax and rental equivalence. The resulting weights are rounded to their nearest part per thousand. This rounding means that there may be some minor differences between the CPI and CPIH group and division indices and percentage changes even when there are no differences in coverage.

1995 CPI estimates

The 1995 COICOP class and group level indices that were calculated for the launch of the HICP are not readily available, so have been estimated. However, weights are available. The class and group indices were calculated by:

1. mapping the items to COICOP division

2. for the locally collected items, where prices are collected from shops around the country, an initial estimate of the formula effect for each item in each month was calculated as the average difference between the CPI and RPI item indices in the following two years, and then applied to the RPI item index for the same month in 1995

3. aggregating the initial item indices to COICOP division using CPI item weights

4. for locally collected items, scaling the initial item indices so that the divisional index is the same as the published CPI index; the same factor is applied to each item index within each division for each month (for centrally compiled items, the formula effect is zero or assumed to be zero, so the CPI item index is the same as the RPI item index)

COICOP amendments

In the original back run for 1988 to 1994, the HICP was classified according to a provisional version of COICOP. This version is very close to the final version but differs at the class level and group level for a handful of indices in divisions 9 and 12 (recreation and culture, and miscellaneous goods and services). The indices in these divisions have been mapped as closely as possible to the final COICOP, but in a very few cases it has not been possible to produce separate estimates at class level. This is the case for:

• 09.4 – Recreational and cultural services

• 09.5 – Books, newspapers and stationery

• 12.3 – Personal effects, not elsewhere classified

The price indices for the COICOP classes in these categories were set equal to their group index.

Additionally, there are some COICOP classes that were not introduced into the CPI until 2000 and later. These are series involving part-payment by the state for the service, or where there are conceptual issues about the construction of the price index. The areas affected by this are:

• 06 – Health (for example, NHS prescription charges, in-patient health care)

• 10 – Education

• 12.4 – Social protection

• 12.6.2 – Financial services (for example, services related to the value of the transaction)

• 12.5.3 – Health insurance

The indices for these services have not been included in the calculations up to 1999 so as to be consistent with the coverage of the CPI during 1996 to 1999.

Notes for: Methodology

1. For more information on how these were calculated, see the Harmonised Index of Consumer Prices: Historical Estimates (PDF, 106KB).

2. For more information on the formula effect see the article Shortcomings of the Retail Prices Index as a measure of inflation.

3. An implied deflator is a measure derived by dividing a current price expenditure series by the volume series. For more information on the implied deflator, see Reconciliation of the differences between the Consumer Price Index and the Implied Price Deflator (PDF, 345KB)

4. The 2014 UK National Accounts were used to be consistent with the rental price index methodology.

5. For more information, see the Consumer Price Indices Technical Manual.

Comparison of CPIH and CPI

The Consumer Prices Index including owner occupiers’ housing costs (CPIH) is the same as the Consumer Prices Index (CPI) with the additional inclusion of rates or Council Tax and owner occupiers’ housing costs (OOH). Figure 3 plots the 12-month growth rate of CPIH and CPI, alongside the difference between the two inflation measures from January 1989 to October 2018.

The figure shows that though CPIH and CPI followed similar paths, the 12-month growth rate in CPIH was higher than that of CPI for most of the period. This pattern changed from around 2003 when the two indices track each other much more closely for the remainder of the modelled series. In the live National Statistic series, the 12-month CPI rate was consistently larger than the 12-month CPIH rate between May 2009 and April 2014. This was also the case in the 15 months to October 2018.

Between January 1989 and December 2004, the 12-month growth rate of CPIH peaked at 9.2% between September and December 1990, seven months before CPI peaked at 8.4%. The largest absolute difference in the annual growth rate of CPIH and CPI during the period occurred in December 1990, with a difference of 1.6 percentage points. The year-on-year CPI rate was higher than the corresponding CPIH rate in the year to March 1992 and again in the 10 months to February 1994. There were further instances in February and March 1995, September and October 2003, and May and June 2004.

There is a large increase in the Council Tax series in April 1990, which is due to the introduction of the community charge. This saw charges rise by 35%. This was reversed the following April. This reversal coincided with an increase in Value Added Tax (VAT), from 15% to 17.5% in April 1991. Whilst the effects of the community charge will only be seen in CPIH, the VAT increase feeds through into both the CPIH and CPI baskets, dampening the community charge reversal.

Over the entire time series from January 1989 to October 2018, CPIH averaged 2.7%, that is, 0.1 percentage points higher than average CPI. Over the modelled period to December 2005, CPIH averaged 3%, that is, 0.3 percentage points higher than CPI. This is larger than the difference in the National Statistic series where CPIH averages 2.3%, that is, 0.1 percentage points lower than CPI, which is 2.4%. When interpreting the long-run series, users should be aware of the limitations of the modelled series, as discussed earlier in this article.

Contributions to the 12-month growth in all-items CPIH

This section analyses the contributions of specific categories of goods and services to the 12-month CPIH growth, discussing them in the context of important economic events over the period.

Figure 4 shows the contributions of each category of goods and services to the overall year-on-year growth rate of CPIH over the period. In all periods, housing, water, gas, electricity and other fuels was the largest driver of changes to the 12-month CPIH rate. However, the extent to which this category has influenced headline CPIH has varied over time in proportional terms. Specifically, the contribution of this category ranged from just under one-quarter (23.9%) in July 1993 to nearly three-quarters (74.6%) in March 2000.

During the economic downturn of the early 1990s, all 12 categories of goods and services contributed positively to the year-on-year CPIH rate. In the months after the recession came to an end, the sterling was depreciating against the Deutsche Mark, failing to keep within the band stipulated by the European Exchange Rate Mechanism. This meant that relatively import intensive categories of goods and services contributed more to the CPIH rate over this period, since depreciation of the pound sterling tends to raise the cost of imports.

Figure 4 shows that apart from housing and utilities, the three largest contributors to the CPIH rate during the recession were the transport, hotels, cafes and restaurants, and food and non-alcoholic beverages categories. The education, and recreation and culture categories made the least contributions to the CPIH rate, contributing an average of 0.70 and 0.37 percentage points respectively over the recession. The contribution of clothing and footwear became negative in April 1992 and remained negative for the rest of the period to December 2004.

The largest contribution of housing, water, gas, electricity and other fuels, in relative terms, was in March 2000 when this category contributed 0.7 percentage points to the headline CPIH rate of 1%. This period was characterised by steady economic growth, with a quarterly gross domestic product (GDP) of 4.2% and strong growth in UK house prices amounting to 16.1%. The contribution of housing and utilities in this period surpassed the combined contributions of the transport, and hotels, cafes and restaurants categories (at 0.4 and 0.3 percentage points respectively). This was offset by negative contributions of the food and non-alcoholic beverages, and clothing and footwear categories (at negative 0.2 and negative 0.3 percentage points respectively).

Figure 5 shows the average contributions for the 1989 to 2005 period compared with the 2006 to 2017 period. Overall, the average contributions from the 1989 to 2005 period were larger than the average contributions from the 2006 to 2017 period consistent with the higher inflation rates experienced in the former period. In both periods, clothing and footwear pulled down CPIH, though this was to a lesser extent in the 2006 to 2017 period, most likely due to the change in price collection guidelines in 2010.

The largest absolute difference in average contributions for the two periods was the housing, water, gas, electricity and other fuels category. The average contribution of this category for the 1989 to 2005 period (1.26 percentage points) was nearly twice as large as the contributions in the 2006 to 2017 period (0.73 percentage points). There were also large increases in the average contribution of education, which more than doubled between the two periods from 0.06 percentage points to 0.13 percentage points. In contrast, the average contributions of the health category had the smallest absolute difference between the two periods.

Though housing, water, gas, electricity and other fuels contributed the most in both periods, the second-largest contributor to CPIH changed between the two periods. In the 1989 to 2005 period, hotels, cafes and restaurants was the second-largest contributor to CPIH, whereas in the 2006 to 2017 period, transport was the second-largest contributor. There were also big changes in the average contributions of communication, which reversed from negative 0.01 percentage points in the 1989 to 2005 period to 0.04 percentage points in the 2006 to 2017 period and was the only category to switch from a negative average contribution to a positive one.

This article provides users with a modelled Consumer Prices Index including owner occupiers’ housing costs (CPIH) historical series back to 1988. We are publishing these estimates as official statistics, recognising that this is a modelled series based on historical data and, as such, should be treated with caution. However, the CPIH series from 2005 onwards remains a National Statistic.

Future work will focus on the development of estimates of CPIH for earlier periods. In January 2018, the National Statistician’s Advisory Panels on Consumer Prices (APCPs) considered the development of this series and recommended publishing from 1988. However, they felt that more work was required for the series between 1947 and 1988. In particular, they suggested exploring household budget surveys as a source of expenditure data for the rental equivalence component, although finding a suitable price indicator for this period is also not straightforward.

We aim to publish the results for the 1947 to 1988 modelled historical series next year, after we have sought further advice from the APCPs. This will provide users with a modelled CPIH historical series of more than 50 years, although the uncertainty associated with estimates is likely to increase the further back in time we go.

Jim O’Donoghue, Onyinye Ezeyi and Christopher Payne.

We are publishing the modelled Consumer Prices Index including owner occupiers’ housing costs (CPIH) historical series at class level and above to give users flexibility to create their own special aggregates. Since the new CPIH historical series is, for the most part, based on the same underlying class level indices as the modelled Consumer Prices Index (CPI) historical series, detailed class level indices for CPI are also available for the first time.

In constructing the CPIH estimates we identified an error in the modelled CPI historical estimates covering the period 1988 to 1996, prior to the introduction of CPI. The error is present at both division and all item levels. However, the CPI National Statistic series, which was introduced in 1997 with the live publication of monthly data, is unaffected. As the historical series was modelled after the event, the data have never been used for any policy or uprating purposes.

When compiling estimates for the CPI for 1988 to 1995, the weights used to aggregate class and group indices to the higher levels of the Classification of Individual Consumption by Purpose (COICOP) hierarchy were not rounded, although the divisional weights were rounded to the nearest part per thousand. With the release of class and group indices, the CPI weights at these levels have now been rounded; the division weights have been left unchanged. The effect of the rounding is to cause some minor revisions in the CPI division percentage changes.

Recalculation of the modelled CPI indices for these previous periods has also resulted in small revisions to the all items CPI historical series, with larger revisions to the 12-month rates for most divisions. As the original CPI estimates were constructed some time ago, it is not possible to identify the cause of the error; however, we have carried out a number of checks to ensure that the new modelled CPIH historical series is correct. We will therefore be revising historic modelled CPI data to align estimates with the CPIH series.

Figure 6 shows the impact of these revisions on the published CPI 12-month growth rates. Of the 96 months between January 1989 and December 1996, there are only 10 impacts (at one decimal place). Of these, seven are downward revisions and three are upward revisions. The maximum revision is 0.2 percentage points in absolute terms, and is due to changes to the July 1991 index. This was one of the four months in the original modelled CPI historical series where indices had to be estimated due to missing or incomplete price data¹. The (absolute) average revision size is 0.01 percentage points.

Download revisions data

Figure 7 shows the impact of revisions on the cumulative growth over the same period. In the index there are 26 revisions, all in an upward direction. The maximum revision is 0.2 index points, and the average size of the revision is 0.03 index points.

Download revisions data

Revisions are larger and more frequent at the division level, with revisions to the 12-month rate of up to 0.7 percentage points in magnitude (except for health, which is discussed later in this section). The revisions occur particularly during 1992 to 1996. The divisions most affected are:

• housing, water, electricity, gas and other fuels, which has 45 revisions of up to 0.7 percentage points in magnitude

• food and non-alcoholic beverages, which has 44 revisions of up to 0.5 percentage points in magnitude

• recreation and culture, which has 53 revisions of up to 0.5 in magnitude

The health division (Figure 18) is something of an exception. There are no revisions to the annual growth rates at all until 1995, which sees a period of large revisions between 0.2 and 0.8 percentage points. Thereafter, 1996 sees larger revisions of between 1.6 and 2.7 percentage points – much larger than are seen elsewhere. Figure 19 shows that revisions of around 1.0 are relatively constant throughout the entire index for health. These are larger than in most other divisions. Therefore, when we calculate annual growth from revised indices there is a limited impact; however, for 1996, we calculate annual growth from revised 1995 indices and non-revised 1996 indices, which generates a larger impact. Moreover, as described in Section 3, many health items were not introduced until after the revised period. This means that estimates are less robust than for other divisions.

Revision charts for each of the 12 COICOP divisions are presented in Figures 8 to 31. The revisions are provided in the data downloads for Figures 6 to 31, and are also available in the dataset accompanying this release.

Download revisions data

Download revisions data

Download revisions data

Download revisions data

Download revisions data

Download revisions data

Download revisions data

Download revisions data

Download revisions data

Download revisions data

Download revisions data

Download revisions data

Download revisions data

Download revisions data

Download revisions data

Download revisions data

Download revisions data

Download revisions data

Download revisions data

Download revisions data

Download revisions data

Download revisions data

Download revisions data

Download revisions data

Notes for: Annex A: Revisions to the Consumer Prices Index, 1988 to 1995

1. For more information, please see the Harmonised Index of Consumer Prices: Historical Estimates (PDF, 106.KB).

The statistics in this release are classified as official statistics. They form part of a longer series for the Consumer Prices Index including owner occupiers’ housing costs (CPIH), of which the latter part, from 2005 onwards, constitutes a National Statistic and is our lead measure of inflation. This modelled series covers the earlier period from 1988 to 2005 and has been constructed using historical data. As an official statistic, the modelled CPIH historical series meets the standards set out in the Code of Practice for Statistics as follows.

Trustworthiness

Our Better Statistics, Better Decisions strategy and business plan are available. This sets out our collective mission to provide high-quality statistics, analysis and advice to help Britain make better decisions, and references our values of honesty, integrity, objectivity, and impartiality, which those working in Office for National Statistics (ONS) are expected to adhere to. It also contains information on our processes and plans to ensure sufficient resources to carry out our statistical functions, and the strategy’s capable objective details our commitment to support the professional capability of ONS staff. This is facilitated by ONS’s Head of Profession for Statistics, who is also responsible for ensuring compliance with the Code of Practice. Our Economic Statistics and Analysis Strategy and Development Plan for Consumer Prices are also available, demonstrating our commitment to being open about our plans and priorities.

Our statistics are published openly online and are available free of charge to all, as set out in our publication policies. How our data are collected, kept and used in line with legislation are also detailed online in our data policies and information charter.

Quality

The methodology used to produce the modelled historical series for CPIH has been developed with input from the National Statistician’s Advisory Panel on Consumer Prices – Technical (APCP-T), and is fully described in the accompanying article. In particular, APCP-T advised that more work should be done to identify appropriate weights data between 1947 and 1987. We have therefore made the decision to publish the series from 1988 to 2005, with a view to publishing the earlier part of the series in 2019. The Consumer Price Inflation Quality and Methodology Information report contains information on the relevance, accuracy, coherence, and comparability of all our consumer price statistics.

For the most part, the series has been constructed from historical Consumer Prices Index (CPI) and Retail Prices Index (RPI) class (or section) level indices and weights, and historical UK National Accounts expenditure data, and so sources cannot be quality assured in the same way as live data. However, most of these series are the same as were used to construct the modelled historical CPI series. Whilst historical RPI data have been used alongside historical CPI data to calculate the additional housing components in CPIH, the formula effect is either not applicable or is demonstrably small for these items (and is certainly smaller than the bias from using available higher-level CPI data, which includes social rents). Information on the quality of all the data sources used in our consumer price inflation measures can be found in our Quality assurance of administrative data used in consumer price inflation statistics publication, and further limitations are described in this article.

These include having to estimate lower-level indices for 1995 and classification differences prior to 1995. In particular, the private rents series used to backcast the measure of owner occupiers’ housing costs (OOH) is based on data from locally collected prices, rather than the administrative dataset used to construct CPIH from 2005 to 2018. This is based on a much smaller sample size and means that the series cannot be adjusted to reflect the composition of the owner-occupied housing market. Analysis suggests that this method is closely correlated with the more robust 2005 to 2013 OOH measure with a slight lead; however, there are some larger differences in the annual growth of up to 2 percentage points in 2006.

Moreover, we would expect the confidence interval around the OOH index to increase the further back in time we go. We have explored using statistical models to improve the quality of the series; however, none of the options considered notably improved the quality of estimates. We have therefore opted to use private rental data directly. Given the historical nature of the data, this remains the most appropriate option for constructing a historical series and is reflected in the decision to label the historical series an official statistic rather than a National Statistic.

Value

Development of a modelled CPIH historical series has been undertaken in response to user needs, and is a high priority on our Consumer prices development plan. Users have provided input throughout. In particular, the views of the National Statistician’s Advisory Panel on Consumer Prices – Stakeholder (APCP-S) and Tetrapartite have been sought. Following methodological advice from APCP-T we consulted with APCP-S and Tetrapartite, and decided to publish the modelled series from 1988 to 2005 whilst continuing to investigate more appropriate data sources for the series from 1947 to 1987. We aim to publish this extended modelled series in 2019. Nevertheless, this remains an improvement on the existing CPIH National Statistics series, which goes back to 2005.

The decision to publish the data at class level was also taken in response to feedback from these groups, and will allow users to construct series that are of particular interest to them. This also means that for the first time, users will have access to a more detailed modelled CPI historical series, which was previously only available at division level.

We have published the modelled CPIH historical series alongside this article, which explains the methodology, and provides some economic context to historical figures. (See also the article Measuring changing prices and costs for consumers and households, which provides clarity on what CPIH measures in relation to our other measures of consumer price inflation.) The article and dataset have been released in accordance with our pre-release access policy. The data will be available as part of the Consumer price inflation time series, to aid re-use of this series. As this is a historical series, it has been constructed entirely using pre-existing historical data.

Finally, there is an inherent trade-off between the pillars of quality and value since the value to users in having access to a longer time series is offset by the greater uncertainty in the historical data. Despite the uncertainty, users have suggested that the modelled historical series is of value, although it is important to remain aware of the limitations of the extended series.

This annex summarises the quality of different models that were explored in developing the back series for Consumer Prices Index (CPI) imputed rents. Two broad approaches were considered. First, treating both Retail Prices Index (RPI) actual rents and CPI imputed rents as stochastic processes in a state-space model¹, borrowing strength for CPI imputed rents in the backcast period from RPI actual rents using correlation in the unobserved components. Second, treating RPI actual rents as deterministic in modelling the relationship between CPI imputed rents and RPI actual rents.

For the period 2008 to 2015, there is graphical evidence that private rentals lead owner occupiers’ housing costs (OOH) growth by four months, though it does not hold as well for the periods 2006 to 2008, and is perhaps mainly driven by aligning the noticeable trough during the recession (Figure 1). The reasons for the differences are not known.

CPI and RPI as stochastic processes

A univariate backcast for such a long period is unlikely to be a good representation of the unknown population process. Therefore, an enhanced model was developed and fitted using the dlm package² in R³ and included estimated covariances in the observation and transition equations for the observation errors, and level and slope errors of the unobserved components. One concern was that the optimisation method had to be changed to include constraints in the maximum likelihood to ensure that correlations were bounded between positive 1 and negative 1. Potentially poorer estimates of the covariance matrices might impact the quality of the back series and estimated confidence intervals. The model was therefore re-fitted using the KFAS package⁴.

The confidence intervals for the estimated back series are very large, which results from a combination of the assumed model in which RPI actual rents are treated as a stochastic process and the fact that the number of time points estimated in the back series is large. Additional improvements could be explored by including data post 2013 in the fitting of the model and attempting to address any estimated discontinuity arising from the change to using Valuation Office Agency data. Other data sources such as mortgage interest payments have been tested but not found to be useful in the models.

CPI as a stochastic and RPI as a deterministic process

Regressing lagged actual rents on imputed rents treats RPI as a deterministic process. One model that could potentially be estimated is simply using RPI actual rents as the historical series. In this situation, a simple linear model would have a co-efficient of one for the variance in RPI actual rents and no intercept. For the models explored here, based on previous analysis it is assumed that RPI is lagged by four months. Model residuals suggest that there is structure left to explain.

There are various ways to address this. One is to use the approach that assumes both series are stochastic and to model the relationship as detailed previously (or other specifications). Retaining RPI as a deterministic variable, regARIMA models⁵ could be used to deal with some of the structure in the model residuals. CPI imputed rent backcasts were explored based on modelling the difference between CPI imputed rents and RPI actual rents with a seasonal regARIMA model of order (0,1,0)(0,0,1) and an automatically identified level shift in January 2009. The main message from this model is that, other than the backcast value for December 2005, all other backcast values are not significantly different from RPI. Therefore, if it is assumed that this model is not misspecified, then there is an argument for simply using RPI actual rents as there is no significant differencebetween CPI imputed and RPI actual rents (lagged by four months).

Summary

Arguments over the best approach are somewhat philosophical based on assumptions about the appropriate model structure. Given the poor accuracy of the model-based backcasts, we prefer to take a data-driven approach and attempt to reconstruct historical imputed rents based on the most appropriate data sources available. The fact that a time series model of the differences between RPI actual and CPI imputed rents finds no strong evidence of a persistent systematic difference lends some support to this approach.

This is not an ideal solution and could be used to argue that no back series should be produced. However, this is overridden by strong user requirements for a back series. If different users were to compute their own back series using different approaches this would lead to problems of comparability in their analyses. Users should therefore be aware of the poor quality of the back series and are encouraged not to use it unless necessary.

Notes for: Annex C: Time series models for creating a historical series for OOH

1. Harvey A (1991), ‘Forecasting, Structural Time Series Models and the Kalman Filter’, Cambridge University Press.

2. Petris G (2010), ‘An R Package for Dynamic Linear Models’, Journal of Statistical Software, Volume 36, Issue 12, pages 1 to 16.

3. R Core Team (2017), ‘R: A language and environment for statistical computing’, R Foundation for Statistical Computing, Vienna, Austria.

4. Helske J (2017), ‘KFAS: Exponential Family State Space Models in R’, Journal of Statistical Software, Volume 78, Issue 10, pages 1 to 39.

5. Findley D, Monsell B, Bell W, Otto M and Chen B (1998), ‘New Capabilities and Methods of the X-12-ARIMA Seasonal-Adjustment Program’, Journal of Business and Economic Statistics, Volume 16, Issue 2, pages 127 to 152.