Developing nowcast methodologies for public service productivity, UK

• We have evaluated and refined our initial nowcast methodology for public service productivity (PSP), in light of 2021 annual data becoming available and a technical review; we previously published experimental PSP nowcasts for 2021 and 2022 in November 2023.

• While nowcasting PSP is challenging, especially considering the volatility of data around the coronavirus (COVID-19) period, the current time lag of two years in producing annual accredited official PSP statistics makes it necessary; this article is a high-level summary of the work to assess and improve on our initial nowcasts.

• We have incorporated a number of technical improvements to produce a refined dynamic regression (DR) nowcast, based on internal and external feedback.

• We evaluated our refined DR method against other nowcasting approaches, specifically compound annual growth rate (CAGR), quarterly annualised growth rate (QAGR), and exponential smoothing (ES).

• Our initial nowcast estimates published in November 2023 were influenced by model parameters chosen, particularly excluding the coronavirus pandemic year 2020, and revisions to the data that fed the model; as a result, the previous nowcasts overestimated growth in 2021 and 2022.

• Our evaluation concludes that, while the refined DR approach produces more accurate estimates than our initial version, a less complex and more transparent QAGR approach performs better at estimating total PSP at the current time. Therefore, the QAGR approach will continue to be used to provide nowcast estimates in our future PSP releases.

Introduction

The Chancellor of the Exchequer asked Professor Sir Ian Diamond, the UK's National Statistician, to undertake a Public Services Productivity Review, in June 2023. To improve measures, the Office for National Statistics (ONS) is partnering with government departments, academics, and expert users to help develop and further improve methodology and data sources.

Built around the pillars of trustworthiness, quality and value, the Code of Practice for Statistics sets standards that producers of official statistics should commit to. The need to balance the timeliness and accuracy of statistics is a long-standing challenge for statisticians, and one that is a key component of the quality pillar. Given the timeliness of our public service productivity source data, which come from administrative sources, our annual accredited official statistics are currently produced at a two-year time lag. As part of the Public Services Productivity Review, we have been working with methodologists to determine how we can produce timelier estimates (nowcast), at a level of accuracy that meets user needs.

Through our development of public service productivity (PSP) nowcasts and release of these as official statistics in development, we are also guided by the Code of Practice for Statistics, which promotes innovation and improvement in the production of statistics. This process has considered the volatility of data around the coronavirus (COVID-19) period, and we continue to seek to improve the accuracy of nowcast estimates. 

This article provides updates on previously published nowcast estimates in our Public service productivity, UK: 1997 to 2022 article. Building on the work to produce 2021 and 2022 estimates, using a dynamic regression approach, this article considers how the 2021 nowcast estimates compared with 2021 accredited official statistics. Dynamic regression is a time series analysis method which uses information from past observations of the series, as well as information from predictor variables (that is relevant annualised quarterly series) to nowcast estimates (more information on this can be found in Section 8: Data sources and quality). 

We also describe methodological changes we have made to produce a new set of estimates for 2022 and 2023, focusing on two separate nowcasting approaches. Finally, we outline issues relating to revisions of the source data that users should consider when gauging the accuracy of these official statistics in development against the more comprehensive accredited official statistics (the next release is due in spring 2025).

Alongside our annual quality adjusted PSP accredited official statistics, available to 2021 in our Public service productivity: total, UK, 2021 article, experimental quarterly estimates of non-quality adjusted PSP are available for the period up to Quarter 2 (Apr to Jun) 2024 in our Public service productivity, quarterly, UK: April to June 2024 bulletin. The nowcast estimates presented here draw data from these quarterly and annual estimates. Nowcasts should be considered experimental and subject to updating and revision. Further details on the concepts and methods used in our PSP statistics can be found in our Sources and methods for public service productivity estimates methodology.

Public service productivity (PSP) nowcast estimates for 2021 and 2022 were published in November 2023 in our Public service productivity, UK: 1997 to 2022 article. Annual, accredited official PSP statistics for 2021 are now available, in our Public service productivity: total, UK, 2021 article, which has allowed us to assess the accuracy of the estimates for 2021.

The accredited official 2021 annual growth estimate for total PSP was 6.5%. This means that our nowcast estimate of 15.1% was a substantial overestimate. Education and healthcare nowcasts also overestimated growth for 2021 while the public order and safety nowcast was closer to the accredited official 2021 annual growth estimate.

Figure 1: Previous nowcast estimates for 2021 substantially overestimated annual growth, except for public order and safety

Total and service area public service productivity, UK, nowcast estimates and accredited official statistics, 2021

Source: Public service productivity from the Office for National Statistics

Download this image Figure 1: Previous nowcast estimates for 2021 substantially overestimated annual growth, except for public order and safety

.png (68.9 kB) .xlsx (16.8 kB)

Given these differences, we have refined our methods and present these in Section 4: Refining the public service productivity nowcasts. 

Previous nowcasts and overestimated growth 

Nowcasting is inherently challenging and estimates produced can be influenced by a number of factors. Sudden events or shocks to the economic system (for example, a financial crash or the coronavirus (COVID-19) pandemic) are difficult to capture accurately in nowcasting models, and many methodologies assume some level of stability over time. This was the most challenging aspect of our previous nowcasts, used to produce 2021 and 2022 estimates in our Public service productivity, UK: 1997 to 2022 article. To address this issue, our dynamic regression methodology, published in November 2023, excluded 2020 data from our estimations, replacing it with a modelled estimate. The model underestimated the effect of COVID-19, which in turn affected the 2021 and 2022 nowcasts.      

Another challenge is the quality of the data used in the model. Our nowcast estimation models rely on proxy indicators derived from quarterly PSP data. While data are similar to the annual series, they are not identical; any divergence between the two will reduce the accuracy of the nowcast estimates (for example, the breadth and granularity of the data or use of a quality adjustment). See our A guide to quality adjustment in public service productivity measures article. Similarly, the underlying data used in the models are also subject to revision, affecting any nowcast estimates produced (see Section 4: Refining the public service productivity nowcasts for further details). 

The type of nowcast method used will also affect the estimates produced. While we chose a dynamic regression approach in our initial nowcasts, there are other methodologies (of varying complexity) that could be used.

Recommendations for improvements 

Given the initial nowcasts significantly overestimated 2021 public service productivity (PSP) growth, we prioritised a series of recommendations from both internal and external experts, including the Office for National Statistics (ONS) Methods and Research Assurance Group (MaRAG) and the Institute for Government (IfG). The following recommendations have been adopted in our latest dynamic regression analysis: 

• Accounting for the uncertainty caused by coronavirus (COVID-19) 

• Greater nowcast granularity 

• Further quality assurance of predictors 

• Comparison with other approaches 

Because of uncertainty around the 2020 data at the time of publishing, our initial set of nowcast estimates excluded 2020 (coronavirus pandemic) from all analyses. The four methodological approaches to improve our nowcast methodology include 2020 within the models, allowing for the variation in 2020 to be modelled. 

Moving away from nowcasting productivity directly, the refined models calculate productivity as a ratio of constituent nowcasts of each of inputs and outputs. 

Technical checks were carried out to assess multicollinearity (highly correlated predictors within the same model), and whether the predictors chosen were qualitatively valid. 

Our reviewers expressed the need to compare a range of alternative approaches. Based on their advice, four approaches were chosen and tested to improve our nowcast methodology:  

• Refined dynamic regression (DR)  

• Quarterly annualised growth rate (QAGR) 

• Compound annual growth rate (CAGR)  

• Exponential smoothing (ES)  

Refined dynamic regression (DR) is a time series analysis method which uses information from past observations in the series and also information from predictor variables. The predictor variables used were the relevant annualised quarterly series. For example, the updated nowcasts for 2021 are a consequence of the observed annualised quarterly series in 2021 and the relationship between the observed annual series and annualised quarterly series in 1997 to 2020. See Section 8: Data sources and quality for more information. The improvements suggested by the expert review were also implemented. 

The quarterly annualised growth rate (QAGR) uses the growth rate in annualised quarterly PSP estimates to produce nowcast estimates. So, for example, the QAGR published in our Public service productivity, quarterly, UK: April to June 2024 bulletin from November 2024 recorded estimated total productivity growth of positive 1.0% in 2022 and negative 0.3% in 2023. The QAGR approach applies the same growth rate to the annual figures. 

The compound annual growth rate (CAGR) calculates the mean annualised growth rate over a set time period. This method smooths out year to year fluctuations by including only the first and last time point in the calculation, so shocks in the intervening time period are not accounted for. CAGR is commonly used for modelling economic data, however, the simplicity of the calculation may not make it the best method for identifying future trends. 

Exponential smoothing (ES) produces a forecast from the weighted average of previous values. This weighting exponentially decreases as the time period is further from the predicted value. This widely used approach is simple, explainable and it makes intuitive sense to give greater weight to closer time points. There are three types of exponential smoothing applicable to annual data, and the best model was identified for each input and output. 

Identifying the optimal approach  

Using time series cross-validation we tested how the four methods would have performed on historic data, nowcasting PSP estimates where sufficient historic data was available (2007 to 2021). The accuracy of each of the four methods was assessed using root mean squared error (RMSE), with a smaller RMSE signifying greater accuracy.  

Figure 2 presents the RMSE for the total and service area nowcast productivity estimates arising from each approach. Based on these, we ruled out CAGR and ES as being insufficiently robust. With the lowest RMSE across all four productivity measures, QAGR showed the greatest accuracy, while the DR approach had a similar level of accuracy for nowcasting total PSP. 

In terms of nowcasting the constituent input and output indices, QAGR also showed the greatest accuracy for three of the input estimates and two of the output estimates. QAGR was the most accurate approach for nowcasting for total inputs, healthcare inputs, education inputs, and outputs for education and public order and safety. DR was the most accurate approach for nowcasting public order and safety input, total output and healthcare output. See our accompanying dataset for equivalent RMSE figures for inputs and outputs.

Two points are worth noting. Firstly, other approaches may have been the best predictor in any one year than the most accurate approach over the 2007 to 2021 period (see Figure 3 for a comparison of QAGR and DR across the time series for total PSP, compared with the official annual estimate). Secondly, as our productivity nowcasts are calculated from constituent input and output nowcasts (output divided by input), there is the possibility of producing a hybrid productivity nowcast and we will consider this as the Public Services Productivity Review continues.

With the cross-validation complete, we compared the improved 2021 nowcasts using QAGR and DR (presented here for the first time) with the accredited official statistics estimates published in March 2024 in our Public service productivity: total, UK, 2021 article  (see Figure 4). Both nowcast approaches estimated total productivity growth much closer to the published official annual figures than previously estimated (DR, 7.1%; QAGR, 7.4%), falling well within the 95% confidence intervals. This is a substantial improvement in the accuracy of our 2021 nowcast.  

Productivity nowcasts for the healthcare and public order and safety service areas also saw improvements using the new nowcasts, with the official published annual figures falling well within the nowcast 95% confidence intervals for healthcare (DR, 12.2%; QAGR, 10.9%; annual figures, 10.8%) and public order and safety (DR, negative 2.3%; QAGR, negative 3.2%; annual figures, negative 3.3%). 

Our new education nowcasts continued to overestimate growth with estimates being substantially above published official statistics (DR, 22.2%; QAGR, 19.0%; annual figures, 13.1%). Further methodological work is currently being undertaken as part of the Public Services Productivity Review to understand the reasons behind the nowcast overestimation within the education service area.

Figure 4: Revised nowcast estimates for 2021 align more closely with accredited official statistics

Total public service productivity and service area public service productivity (education, healthcare, and public order and safety), official statistics estimates and improved nowcast estimates, 2021, UK

Source: Public service productivity from the Office for National Statistics

Download this image Figure 4: Revised nowcast estimates for 2021 align more closely with accredited official statistics

.png (67.9 kB) .xlsx (16.5 kB)

This section presents 2022 and 2023 nowcast estimates for total public service productivity (PSP), inputs and outputs using the quarterly annualised growth rate (QAGR) and refined dynamic regression (DR) approaches.  

Total public service productivity nowcasts are presented in Figure 5 (further methodological work is required to produce nowcasts by service area). Figure 5 shows estimated negative growth of 0.7% in 2022 using the DR method (95% confidence interval, negative 2.6%, positive 1.4%) and positive growth of 1.0% using QAGR (95% confidence interval, negative 1.0%, positive 3.1%).

Estimated productivity growth in 2023 was negative 0.2% using dynamic regression (95% confidence intervaI, negative 2.9%, positive 2.5%) and negative 0.3% using QAGR (95% confidence interval, negative 3.1%, positive 2.6%). Given that confidence intervals for QAGR and DR include zero, we are unable to predict with confidence whether there will be growth or contraction in 2022 and 2023.

Nowcast estimates for total public service inputs and outputs are shown in Table 1.

Impact of revisions to the data for 2022 and 2023 nowcasts 

When producing nowcast estimates we use the most up-to-date data, however, both the annual and annualised quarterly data we use to derive nowcasts are subject to revision; for example, because of revisions made to data in the UK National Accounts, The Blue Book. This means that with each release of revised data, our nowcast estimates will also be revised. To illustrate the effect of revisions to the underlying data on our nowcast estimates, Table 2 presents how the 2022 and 2023 nowcasts have changed using data published from the three most recent quarterly PSP releases. Following the Quarter 4 (Oct to Dec) 2023 release (published 3 May 2024), revisions have reduced nowcasted year-on-year growth rates in each subsequent quarterly release, across both methods.

Additionally, users should be aware that revisions are made to the data in every annual total public services productivity release. We anticipate notable methodological changes, and therefore revisions in the next annual release, because of improvements arising from the public services productivity review.

Annualised quarterly data 

The annualised quarterly data series is an aggregated annual estimate derived from non-seasonally adjusted quarterly estimates, using arithmetic averaging. 

Classification of the functions of government 

The classification of the functions of government (COFOG) is the structure used to classify government activities. It is defined by the United Nations Statistics Division. 

Confidence interval 

Confidence intervals use the standard error to derive a range in which we think the true value is likely to lie. 

Nowcast 

Nowcasting in economics involves predicting (currently unavailable) economic estimates from the present, the recent past, or the near future. Nowcasting uses economic data for previous years, and often other relevant economic variables to produce estimates.  

Public services 

These are services delivered by or paid for by government (central or local). If paid for by the government, they may be delivered by a private body. For example, the provision of nursery places by the private sector, where these places were funded by the government. 

Quality adjustment 

A statistical estimate of the change in the quality of a public service, using an appropriate metric, such as educational attainment in schools as part of the education quality adjustment, and safety in prisons as part of the public order and safety adjustment. 

Service area 

The way we refer to the breakdown of public services into nine areas, closely following COFOG.

Regression methods  

The nowcasts are produced using dynamic regression (DR) and the relevant annualised quarterly series. DR is a time series analysis method which allows nowcasts to use information from past observations of the series and also information from predictor variables. The predictor variables used were the relevant annualised quarterly series. For example, total output (annual) is regressed on total output (annualised quarterly).

Therefore, the nowcasts are a consequence of the observed annualised quarterly series in 2022 and 2023 and the relationship between the observed annual series and annualised quarterly series in 1997 to 2021 (2020 was included in the revised models). DR is an extension of autoregressive integrated moving average (ARIMA) modelling; the extension allows inclusion of predictor variables. 

For example, a standard regression would be:

In this equation, the error is assumed to be uncorrelated. However, in DR, the error term is assumed to follow an ARIMA process.

For example, if the error term followed an ARIMA process the model would be:

This is where the final error term (epsilon) is assumed to be white noise. 

These equations have been reproduced from "Forecasting: Principles and Practice" (third edition) by Rob J Hyndman and George Athanasopoulos.  

Official statistics in development  

These statistics are labelled as "official statistics in development". Until September 2023, these were called "experimental statistics". Read more about the change in our Guide to official statistics in development.  

These statistics are based on information from annual and quarterly public service productivity estimates. We are developing how we collect and produce the data to improve the quality of these statistics. Read more in our Public Services Productivity Review. 

Once the developments are complete, we will review the statistics with the Statistics Head of Profession. We will decide whether the statistics are of sufficient quality and value to be published as official statistics, or whether further development is needed. Production may be stopped if they are not of sufficient quality or value. Users will be informed of the outcome and any changes. 

We value your feedback on these statistics. Contact us at psp.review@ons.gov.uk.

The aim of this article is to set out work we have done to develop our nowcasting methodology to produce more timely estimates of annual public service productivity (PSP) indicators, ahead of the publication of accredited official statistics.  

We have highlighted: 

• improvements that we have made to our initial estimates through incorporating further methodological advice 

• how we have tested the robustness of our methodology through the cross-validation of four different approaches 

• that the refined dynamic regression (DR) and quarterly annualised growth rate (QAGR) produced the most accurate estimates when compared with the actual annual accredited national statistics 

• estimates for 2022 and 2023 total PSP, inputs and outputs using DR and QAGR approaches 

It is clear from our cross-validation analysis that the refined DR method provides more accurate nowcast estimates than our initial DR approach. However, the simpler and more easily understood QAGR approach performs better than DR at estimating total PSP and PSP in the healthcare, education and public order and safety service areas. 

As a result, we conclude that, at this stage, the QAGR approach should continue to be used to provide nowcast estimates in our future PSP releases. We intend to review the performance of the two optimal methodologies identified here (DR and QAGR) on an annual basis to ensure this conclusion remains valid. Future developmental work may also include exploring other nowcast methodologies such as a hybrid approach or state-space models.