Comparing ONS’s economic data with IHS Markit and CIPS Purchasing Managers' Index surveys

• This article investigates the coherence of Office for National Statistics (ONS) Monthly Business Survey (MBS) data and the corresponding Purchasing Managers’ Index® (PMI®) surveys from IHS Markit and the Chartered Institute of Procurement and Supply (CIPS).

• PMIs show the strongest relationship with ONS three-month on three-month a year ago data, with significant and sustained correlations found between the two datasets.

• Despite asking about month-to-month variations, PMIs have no significant correlation with ONS month-on-month MBS diffusion indices or official estimates of growth.

• This may suggest that PMI respondents take a wider and longer-term view of business conditions than just month-to-month variations as well as allowing for seasonal and other distorting factors.

• Even when official estimates and diffusion indices share the same base data, the difference in methodologies can produce two series with no significant correlation.

• Diffusion indices created from the MBS show similar tendencies to over or underestimate during economic shocks that are seen in the PMIs, though the rarity of economic shocks means that the sample size for this is small.

The official estimates of monthly output in the construction, manufacturing and services industry releases inform users of the state of the economy. Important users such as policymakers and forecasters are also interested in timelier indicators of economic performance, such as the Purchasing Managers’ Index (PMI), which can be used to extract an earlier signal on the UK economy. The PMIs are reported monthly but released earlier than official estimates, providing one of the first indications of UK economic performance. Recent innovations also include faster indicators of economic activity based on Value Added Tax (VAT) returns.

Because of the different methodologies used in official estimates of output and in diffusion indices, these indicators can show different results for the performance of the various sectors in the UK economy. This article seeks to build upon work done previously to investigate the statistical coherence between official estimates and diffusion indicators such as PMIs.

Office for National Statistics (ONS) data are regularly reviewed as part of annual updates and reviews of both sources and outputs for consistency. This can lead to revisions to the published estimates. The different vintages of data may have different properties and earlier vintages may have slightly different levels of correspondence with the IHS Markit PMIs. The latest data we have available were used as part of this analysis.

Table 1 summarises the main difference between published Office for National Statistics (ONS) outputs and IHS Markit Purchasing Managers’ Index (PMI) data, more information can be found on Markit’s website.

Constructing the diffusion indices

For the purposes of this analysis, diffusion indices comparable with the Purchasing Managers’ Index (PMI) have been created from a subset of the Monthly Business Survey (MBS) data. A form of standardisation is then applied to both sets of diffusion indices and to official estimates of growth in gross domestic product (GDP) and the services, manufacturing and construction sectors to enable direct comparison of the different indicators. There are four steps to create diffusion indices.

Step 1

The first step in creating the diffusion index is to adjust the Office for National Statistics’s (ONS’s) sector coverage so that it matches, as closely as possible, that of IHS Markit’s. This is done simply by dropping the businesses with the Standard Industrial Classification (SIC) codes excluded from IHS Markit from the Monthly Business Survey (MBS) data. This affects the services sector primarily. It should also be noted that industries that are not included in the MBS are not included in the diffusion index.

Step 2

The turnover for each individual business is then compared with its turnover from the previous period. Where businesses have entered or left the ONS sample, they are not included in the diffusion index. It should be noted that this could introduce a bias over time, as poorly performing business are more likely to go out of business and so be excluded and not replaced in the sample.

The percentage change in turnover compared with the preceding period is calculated and is compared with a user-defined threshold value. The purpose of this threshold value is to capture the fact that in practice the IHS Markit PMI will not be sensitive to small changes in business activity and so will likely report no change even if it has actually slightly increased or decreased.

A plus or minus 5% threshold has been chosen for the analysis in this report. A comparison of the effects of different threshold levels can be found in the appendix in Figures 19 and 20 where it is demonstrated that the choice of threshold level has little effect on the level of correlation found with other series. If the percentage change in turnover for a business is greater than the threshold then it is classified as having increased business activity, if its percentage change falls between the threshold values then it is classified as having the same business activity and if its value is below the threshold value then it is classified as having the reduced business activity.

Step 3

The diffusion index is then calculated using the following formula:

where I is the value of the index, P is the percentage of businesses and the subscripts U, S and D represent the possible directions of business activity compared with the previous month of “up”, “same” and “down”. By multiplying each by the appropriate weighting factor this gives a diffusion index where a value above 50 indicates growth, a value of 50 indicates no change and a value below 50 indicates contraction.

Step 4

Following this the diffusion index is seasonally adjusted, where appropriate, using X13 ARIMA-SEATS software.

For comparison purposes, the PMI, diffusion indices formed from the MBS and official ONS estimates of growth are transformed into standardised units. This is done by calculating the mean and standard deviation for each series and then subtracting the mean from each value in the series before dividing by the standard deviation. This means that all three series will have a mean of 0 and standard deviation of 1, making it easier to determine how consistent they are with each other.

It should be noted that as this process sets the mean of each series to zero, rather than the no change values of 50 for the diffusion indices and 0 for the index growths, values just above and below 0 on the shifted series cannot necessarily be simply interpreted as a prediction of growth or contraction.

IHS Markit and CIPS also publish a composite, all sector PMI, which is constructed by weighting each of the individual sector PMIs by the weight of that sector’s contribution to UK GDP. We compare a whole sector MBS diffusion index and GDP growths with this all sector PMI, which we build by weighting each of the three individual sector PMIs as described by IHS Markit.

Measuring correlation between time series

Many of the time series considered in this article are non-stationary, which means that their statistical properties such as their mean, variance or autocorrelation change over time. This makes the traditional method of assessing correlations, the Pearson correlation coefficient, unsuitable as when applied to non-stationary series it can indicate spurious correlations rather than actual relationships between the two series.

An alternative way to measure the level of correlation between two non-stationary time series has been proposed by Gilney Zebende, the detrended cross-correlation analysis (DCCA) coefficient – which is the method adopted in this analysis. The coefficient is found by constructing a profile, or cumulative sum of the time series minus its mean. This profile is then divided up into a number of overlapping boxes of length scale or window size s, which is a property chosen by the analyst. Within each box a linear fit of the time trend is constructed and the residuals from the linear fit and the profile are used to calculate the detrended variance and detrended covariance for each box. The detrended variance and covariance are then averaged over all boxes and these are then used to calculate the correlation coefficient in the normal way. A full derivation of the DCCA coefficient can be found in Ladislav Kristoufek’s Measuring correlations between non-stationary series with DCCAcoefficient paper.

The DCCA coefficient is a function of the chosen length scale, s, which allows for the strength of correlation to be tested across different time scales by varying s. The coefficient ranges from negative 1 to 1, with negative 1 indicating perfect negative correlation, 0 indicating no correlation and 1 indicating perfect positive correlation. The minimum time scale considered, or the minimum value of s, must be larger than s = n +2, where n is the order of the polynomial fitted to each box as described by Yuan and others. As we have used a linear fit, n = 1 so we consider only values of s greater than 3 in this article. We also set the maximum value of s = T/5 where T is the length of the time series considered, as is standard in the literature according to Kristoufek (PDF, 449KB).

Finally, to find confidence intervals around zero correlation we generate 10,000 pairs of white noise series with no correlation between them. The DCCA coefficient between each is calculated and together they are used to fit a normal distribution truncated in the range [negative 1, 1], which is then used to find the 95% confidence level for each time series length T and window size s considered.

The different methodologies of official estimates of output and diffusion indices can lead to very different conclusions about economic performance even when based upon the same underlying data. This can be demonstrated with the short example in Table 2, which shows the output of four fictional businesses in January and February and their corresponding growth rates.

As Table 2 shows, official estimates based on these four businesses would indicate an overall negative 2.4% growth rate. In contrast, diffusion indices simply register whether output has increased, decreased or stayed the same, so if this data were used to construct a diffusion index it would be interpreted as in Table 3.

Applying the formula for diffusion indices shown in the methodology section gives:

with a value of 62.5 larger than the no change level of 50 and indicating growth in the sector unlike the official estimates, which show a 2.4% contraction.

This example shows how, even when diffusion indices share the same base data as official estimates of output, the two indicators can give very different conclusions about economic performance. The primary cause of this is that diffusion indices, unlike official estimates, do not account for the magnitude of changes in output in individual respondents, so while there are two businesses, C and D, whose output has increased, this is not enough to compensate for the larger decrease seen in business A. This is an effect that diminishes with increased sample size.

Previous articles comparing Office for National Statistics (ONS) data with IHS Markit’s Purchasing Managers’ Indices (PMIs) found the strongest relationship between the two to have resulted from comparison of ONS three-months on three-months a year ago data with PMIs.

The figures in this section show the IHS Markit PMIs and diffusion indices constructed from the Monthly Business Survey (MBS) as line graphs compared with official estimates of growth shown as the bar graphs, all in standardised units.

The match between the three-month on a year ago MBS-based diffusion indices and the PMIs is strong, with the two measures closely tracking each other across the period considered in all three sectors. Unlike comparisons with ONS month-on-month or three-month on three-month data, the two diffusion indices show similar levels of volatility and there are no prolonged periods of discrepancy between the two across any of the sectors.

The relationship between official estimates and the two diffusion indices is also reasonably strong as the three indicators generally move together across the three sectors individually and in the all sector measure. Only one period of sustained discrepancy exists. This is in the services sector from 2012 to 2014 where the two diffusion indices predict a period of below average growth followed by one of above average, while official estimates show exactly the opposite.

Though there are differences in base data between official estimates and the two diffusion indices because of the IHS Markit PMIs having less coverage of the services sector, this period of discrepancy remains, even when an MBS diffusion index is generated using the full ONS MBS services sector coverage. This indicates that this period of disagreement is not due to differences in sector coverage. It should also be noted that prior vintages of ONS data are generally more closely aligned with the PMIs.

Finally, in all three sectors the PMIs indicate that the economic performance across three sectors just after the Brexit referendum, in July 2016, was substantially weaker than either the MBS-based diffusion index or official estimates predicted. It should be noted that this could be affected by the fact that PMIs ask respondents to compare the situation mid-month with a month ago, rather than looking at the whole month as ONS surveys do. Further work on the performance of all three indicators during economic shocks can be found in the Indicators during economic shocks section.

Figure 5 shows the detrended cross-correlation analysis (DCCA) cross-correlation coefficients between the MBS-based diffusion indices and IHS Markit PMIs for all three sectors. Marked in black on the graph is the 95% confidence level; we take correlations to be statistically significant if the DCCA coefficients are larger than the 95% confidence level, that is if they lie above the black line.

Figure 5 shows that there is a strong, positive correlation between the two diffusion indices across all three sectors and for the UK economy as a whole, with the correlation becoming stronger as longer time periods are considered.

The correlations become significant at the 95% level when sections of the two series longer than five, six and nine months’ worth of data are compared for the construction, manufacturing and whole economy, and services sectors respectively. This means that between the manufacturing PMI and manufacturing MBS diffusion index there will be a significant correlation when comparing the data from both over the period from January to June (or any other six-month period or longer), while for the services sector you would need to compare data from January to September (or any other nine-month period or longer) for there to be a significant correlation.

As previously noted, for the construction and manufacturing sectors and the whole economy indicators, the two diffusion indices correspond well with the official estimates of output. However, this is not the case for the services sector, where from early 2012 to 2014 there is a period of discrepancy between the two diffusion indices and official estimates. This difference in congruence between the services sector and the other sections of the economy is clearly shown in the DCCA coefficients in Figure 6. As with Figure 5, correlations are taken to be significant if they lie above the upper 95% confidence bound. A lower 95% confidence bound also exists for negative correlation, this is not shown as no pair of series is significantly negatively correlated at the 95% level.

Figure 6 shows that there is a strong, positive correlation significant at the 95% level between both the MBS-based diffusion indices and PMIs, with ONS official estimates of growths for all window sizes considered for the manufacturing and construction sectors and all sector measure. This can be seen from the fact that the DCCA coefficients for these sectors lie above the upper 95% confidence limit. Significant correlations are also found between growths in gross domestic product (GDP) and both all sector diffusion indices, with PMIs interestingly showing a stronger correlation with official estimates of GDP growth than the MBS-based diffusion index.

However, for the services sector, neither the MBS-based diffusion index nor the PMIs have any significant correlation with headline services three-month on year growths for any window size as can be seen from the fact that their DCCA coefficients lie below the upper 95% confidence interval. This lack of any significant relationship in the services sector may be because there appear to be two distinct periods, from 2012 to 2014, where the diffusion indices and official estimates move in opposite directions and from 2014 onwards, where the diffusion indices and official estimates loosely track each other.

This period of discrepancy and lack of any significant correlation with Index of Services’ growths remains even when a diffusion index is constructed using the same data as the Index of Services (without non-MBS industries), rather than the subset used for the previous indices that match sector coverage with IHS Markit and CIPS. This, and the very good match between the ONS diffusion indices and the PMIs precludes differences in the base data from being responsible for this discrepancy and suggests it may be the differences in methodology between diffusion indices generally and the ONS’s official estimates of output that are the cause. Regardless, neither the MBS- based diffusion index nor the PMIs are a good indicator for three-months on three-months a year ago movements in official estimates of services output despite the results found for the other sectors.

Overall there is a positive correlation between ONS three-month on three-month a year ago MBS-based diffusion indices and PMIs that is significant at the 95% confidence level across a wide range of medium- to long-term time spans for all three sectors and for the all sector measure of the UK economy. For the construction sector, manufacturing sector and all sector measures, significant correlations also exist between both the MBS-based three-month on year diffusion indices and PMIs with the three-month on year official estimates of growth. However no such relationship exists with services headline growths and either the MBS-based diffusion index or PMI.

The strength of the relationships between ONS three-month on year data and PMIs is stronger than that with either ONS month-on-month or three-month on year data. This suggests that PMI respondents take a wider view of business conditions when replying to PMI surveys than just month-to-month changes, possibly also comparing with performance in the same month a year ago.

The graphs for this section are presented in the appendix as Figures 7 to 12. The plots of the three series show that the Monthly Business Survey (MBS)-based diffusion indices remain more volatile than the Purchasing Managers’ Indices (PMIs), however, unlike with the month-on-month data the two series do seem to track each other.

The detrended cross-correlation analysis (DCCA) coefficients, shown in Figure 11, support this and indicate that for the services and construction sectors and all sector indices there is a significant correlation between the MBS-based three-month on three-month diffusion indices and the PMIs when periods longer than nine and six months respectively are analysed, though no significant correlation is found between the two indicators for the manufacturing sector.

All four of the MBS-based diffusion indices have significant correlations with official estimates of three-month on three-month growths as can be seen in Figure 12, though for the services sector this is only for periods of 4 to 11 months. Between the PMIs and Office for National Statistics (ONS) official estimates of growth, significant correlations are only found for the manufacturing and all sector measure when periods of six months or longer are considered.

Across all three sectors there are varying relationships between ONS three-month on three-month data and PMIs. Both services and construction show significant correlation between the MBS-based diffusion index and PMIs but no relationship between PMIs and official estimates of growth. In contrast, manufacturing has no significant correlation between its diffusion index and PMIs but is the only sector with significant correlation between PMIs and official estimates of growth. The all sector PMI is the only one to show significant correlation with both the MBS-based diffusion index and official estimates of growth.

The graphs for this section are presented in the appendix as Figures 13 to 18. The plots of the three series show that across all three sectors, both the month-on-month headline growths and Monthly Business Survey (MBS)-based diffusion indices are substantially more volatile than the Purchasing Managers’ Indices (PMIs) and do not indicate the sustained periods of above and below average growth seen in the PMIs.

The detrended cross-correlation analysis (DCCA) coefficients, shown in Figure 17, indicate that there is no significant correlation between the MBS-based month-on-month diffusion indices and the IHS Markit PMIs, with construction only above the confidence threshold when periods of four months are considered and dropping below for all longer periods. As this correlation is not sustained across multiple window sizes we do not take this as evidence of a significant correlation between the two series.

Between the diffusion indices and official estimates of month-on-month growths, seen in Figure 18, significant correlations are found between the MBS-based month-on-month services, construction and all sector diffusion indices and their respective official estimates of growths. There is no significant correlation between any of the IHS Markit PMIs and any of the Office for National Statistics’s (ONS’s) official estimates of month-on-month growths. Overall, there is no significant relationship between official estimates of month-on-month growth and PMIs or between diffusion indices constructed from MBS data and PMIs.

The following summary tables show whether significant, sustained correlation was found between the Markit Purchasing Managers’ Indices (PMIs) and all Office for National Statistics (ONS) series and between official estimates of growth and both Markit PMIs and Monthly Business Survey (MBS) diffusion indices. To avoid counting spurious correlations we define two series as having sustained correlation if they have significant correlation at the 95% level across at least three or more consecutive interval lengths.

Previous work comparing official estimates of growth with Purchasing Managers’ Indices (PMIs) has noted that: “with unexpected and prolonged shocks, the IHS Markit/CIPS trend exaggerates the magnitude; but with expected/short ‘shocks’, IHS Markit/CIPS under-estimates”. This work did not include the construction of a diffusion index from Office for National Statistics (ONS) Monthly Business Survey (MBS) data and so the effects of sudden shocks on MBS diffusion indices could not be investigated.

Two possible examples of economic shocks have been identified. The first was in June 2012 when the Queen’s Diamond Jubilee meant that there were extra holidays; this can be considered an example of an expected shock as the disruption caused by the Jubilee was known in advance.

The second was the EU referendum result on 23 June 2016; previous HM Treasury analysis found that a vote to leave would “cause a profound economic shock” and so this is taken as an example of an unexpected shock. Because of the fact the referendum took place late in June, we look at July 2016 to see its effects.

Though no significant correlation was found between either official estimates of month-on-month growth and PMIs, or month-on-month diffusion indices constructed from MBS data and PMIs, it is still instructive to look at how both indicators react to sudden shocks. This can be done by looking at Table 6, where the value of each indicator in July 2016 can be found in units of standard deviations from the mean.

Both diffusion indicators indicate that July 2016 was a significantly below average month, with all eight readings more than one standard deviation below the mean. The PMIs estimates are further below average than the MBS-based diffusion index for all three sectors of the UK economy, with the difference between the two estimations for services and GDP substantially larger than for the other two sectors.

In contrast, official estimates indicate that July 2016 was an average month for output, with all three sectors growths’ within one standard deviation of the mean. These results corroborate previous findings that, with unexpected shocks, PMIs tend to exaggerate the effect relative to official estimates but show that the same is also true of the MBS-based diffusion index. This may indicate that the effect of the shock was more diffuse and had a small effect on many firms, or that it impacted relatively more upon smaller firms. As the magnitude of any decrease in performance are not accounted for by diffusion indices, these effects could lead them to overestimate the impact relative to official estimates.

The performance of the three indicators during the expected shock of the Queen’s Diamond Jubilee was very different. Both diffusion indices estimate that June 2012 was a substantially below average month, however, aside from the MBS-based construction diffusion index, neither comes close to the official estimates of how far below average the month was.

This supports previous findings that PMIs tend to underestimate the effect of expected shocks relative to official estimates but finds that MBS-based diffusion indices also underestimate the effect of shocks, even when based upon the same data as official estimates.

A possible explanation for this is that larger firms were affected more by this shock than smaller firms, as the relative sizes are not accounted for by diffusion indices. In the case of PMIs it could also be the case that firms make allowances for the unusual trading pattern. That the MBS-based construction diffusion index is much lower and closer to official estimates than the other two sectors suggests that the effect in the construction sector was more diffuse than elsewhere.

The behaviour of the indicators during these two shocks is consistent with what was previously found. PMIs overestimate the effect of unexpected shocks and underestimate the effects of expected ones relative to official estimates. However, it is also found that diffusion indices formed from the same data as official estimates have the same tendencies, though to a lesser degree. This suggests that the tendency for the two indicators to under and overestimate relative to official estimates is a consequence of the properties of diffusion indices, particularly in the case of the MBS-based ONS diffusion indices where, unlike PMIs, businesses are not weighted by size.

Statistically significant relationships have been found between IHS Markit’s Purchasing Managers’ Indices (PMIs) and Office for National Statistics (ONS) data.

By comparing PMIs with a diffusion index calculated using Monthly Business Survey (MBS) data based on three-month on three-month a year ago changes, significant and positive correlations were found between the indicators for all three individual sectors and for the whole economy diffusion indices across time periods of a few months or longer.

Significant correlation was also found between both the MBS-based diffusion indices and PMIs with official estimates of growth for the manufacturing and construction sectors and the whole economy measures, but not for services where the two diffusion indices do not closely track official estimates of output.

Relationships were also found to exist between three-month on three-month MBS-based diffusion indices and official estimates of three-month on three-month growth with PMIs. For the services and construction sectors and all sector measure, a significant positive correlation was found between the MBS-based diffusion index and the PMIs. While this was not found for the manufacturing sector, the manufacturing PMI and all sector PMI did show significant, sustained correlation with ONS official estimates of three-month on three-month growths. The utility of diffusion indices as predictors of three-month on three-month changes is also illustrated by the fact that all four MBS-based diffusion indices are significantly correlated with official estimates of growth over a range of time periods.

No significant relationships were found to exist between the PMIs and either official estimates of month-on-month growth or the month-on-month MBS-based diffusion indices. Both the ONS MBS-based diffusion indices and official estimates of growth were more volatile than the PMIs and no significant correlation was found between either and the PMIs.

Statistically significant correlations were found between the services, construction and all sector MBS-based diffusion indices and official estimates of month-on-month growths, though these correlations were substantially weaker than those found in the three-month on three-month a year ago.

The limitations of diffusion indices for predicting the more volatile month-on-month changes in output is illustrated by the lack of any significant correlation between the manufacturing diffusion index and growths, despite the two measures being based upon the same underlying data. On the evidence of this analysis, the IHS Markit PMIs cannot be taken as an early indicator for official estimates of month-to-month changes in output.

Finally, previous findings that the IHS Markit PMIs overestimate the effect of sudden economic shocks and underestimate the effect of shocks known in advance relative to official estimates of output were corroborated, though the sample size for this is small. However, the role of PMI responder sentiment as a potential cause of this is called into question by the fact that the MBS-based diffusion indices exhibit similar over and underestimation relative to official outputs. That these effects occur even when the diffusion index and official estimates share base data suggests that these effects may arise from limitations with diffusion indices, such as the fact that the magnitude of changes for individual businesses is not used to form the indicator.

Henry Duquemin,Mark Stephens, Office for National Statistics.

Threshold comparison

Sector coverage differences

ONS Services coverage: SICs 45000 to 97000

IHS Markit PMI Services coverage: SICs 49100 to 64999 ; 68100 to 82990 ; 90000 to 96090

ONS Manufacturing coverage: SICs 10100 to 33200 (excluding 19100-19209 and 24100-24340)

IHS Markit PMI Manufacturing coverage: SICs 10100 to 33200 ; 38300 to 38320 ; 58100 to 58190 ; 59200 to 59200

ONS Construction coverage: SICs 41000 to 43999

IHS Markit PMI Construction coverage: SICs 41000 to 43999