In November 2017, Office for National Statistics (ONS) suspended Dataset A08 (Labour market status of disabled people) for subsequent time periods due to an apparent discontinuity between Quarter 2 (Apr to June) 2017 and Quarter 3 (July to Sept) 2017. While the evidence was not conclusive we could not identify any quality issues and reinstated the dataset from 15 May 2018 subject to the health warnings set out later in these points.
We investigated the potential reasons for the change under four main groupings:
change in survey instrument
change in respondent behaviour (for example, public awareness and or campaigns)
error in processing
The analysis presented in this article rules out the first and fourth groupings and although it suggests some contribution from the second grouping we will continue looking into the third grouping although further analysis on the impact of sampling volatility has not been conclusive.
The reduction in size of the apparent discontinuity in the number of people reporting disabilities1, seen in the Quarter 3 2017 Labour Force Survey (LFS), reversed in Quarter 2 2018 for the harmonised standard definition of disabled people2 The comparisons in this article have been made using not seasonally adjusted data so some of the change could be attributed to seasonal variation.
Although the increase began suddenly at the start of an LFS quarter, analysis of the LFS design did not identify any cause that could have introduced a change in the pattern of disability reporting in Quarter 3 2017; increases were seen across most response types and all of the interview modes. In Quarter 2 2018, there was an increase in personal responses and responses brought forward but none of these increases were unprecedented. There were increases across both interview modes but more prevalent in face-to-face interviews, which is also not unprecedented.
Analysis of the LFS suggests that the increases in the number of disabled people were generally broad-based in Quarter 3 2017. Increases were more concentrated in the employment and inactivity categories in both Quarter 3 2017 and Quarter 2 2018. In Quarter 3 2017, there were increases across all age bands and in the proportion of respondents reporting disability across all waves except wave 2. In Quarter 2 2018, there were increases in waves 1, 2 and 5 but wave 1 and 5 stand out particularly when compared with other waves and among people aged 25 to 34, 35 to 49 and 50 to 64 years but the increases are large only for the 25 to 34- and 50 to 64-year-olds.
As a result of the apparent discontinuity, the volatility in the series and the inconclusive investigations at this stage, comparisons should be treated with caution between Quarter 2 2017 and subsequent quarters. Users should be mindful that the data are not seasonally adjusted so any change that they are seeing between consecutive quarters could be due to seasonality. Furthermore, any short-term changes may be subject to sampling variation; for example, the 95% confidence interval around the number of disabled people in Quarter 2 2018 (7.5 million) is between 7.3 million and 7.6 million.
Users can continue making comparisons prior to the apparent discontinuity, between a given quarter and the same quarter in previous years (we normally advise users to make these “same quarter” comparisons when using not seasonally adjusted datasets).
Comparisons between periods following the apparent discontinuity should also be treated with caution while investigations continue. Further recommendations on “same quarter” comparisons after the apparent discontinuity will be given in November 2018 when we publish the data for Quarter 3 2018 and we have a whole year of data after the apparent discontinuity. The volatility in the data we see on a quarterly basis reinforces the need to wait for more data before a recommendation is made.
Notes for: Main points
In the Labour Force Survey (LFS) respondents self-identify themselves as disabled or not disabled.
The Government Statistical Service (GSS) Harmonised Standards focus on a “core” definition of people whose condition currently limits their activity. In summary the core definition covers people who report:
(current) physical or mental health condition(s) or illnesses lasting or expected to last 12 months or more
the condition(s) or illness(es) reduce their ability to carry out day-to-day activities
In November 2017, Office for National Statistics (ONS) suspended publication of disability employment figures based on the Labour Force Survey (LFS) and the Annual Population Survey (APS), due to an apparent discontinuity between Quarter 2 (Apr to June) 2017 and Quarter 3 (July to Sept) 2017. Working with stakeholders such as the Department for Work and Pensions (DWP), we conducted a review of the datasets to determine why the step change happened.
While the analysis did not result in a conclusive reason being identified, the dataset was reinstated in May 2018 alongside the publication of the Analysis of the discontinuity in the Labour Force Survey disability data: April to June 2017 to July to September 2017 article to share the investigations, conclusions and next steps. This article updates the analysis of the disability employment estimates three months since their reinstatement.Nôl i'r tabl cynnwys
There has been a steady increase in the UK population aged 16 to 64 years over the last few years, averaging around 150,000 more people per year. Figure 1a shows an increase of 130,000 in the number of people aged 16 to 64 years reporting a disability under the harmonised standard definition (disabled people) between Quarter 1 (Jan to Mar) 2018 and Quarter 2 (Apr to June) 2018. Over the same period, there was a fall of 111,000 in the number of people aged 16 to 64 years who are not disabled according to the harmonised standard, either because they have stated that they are not disabled or their condition does not meet the Government Statistical Service (GSS) harmonised definition (not disabled) (Figure 1b).
The increase in the disabled population of 130,000 is not unprecedented, with bigger increases taking place prior to the apparent discontinuity between Quarter 2 2017 and Quarter 3 (July to Sept) 2017. Moreover, a decrease in the non-disabled population of a similar magnitude has also taken place prior to the apparent discontinuity.
Both the increase in the disabled population and the fall in the non-disabled population are smaller in magnitude than the increase (391,000) in the disabled population and the decrease (363,000) in the non-disabled population between Quarter 2 2017 and Quarter 3 2017. However, these quarterly changes returned the series to the level seen in Quarter 3 2017. As mentioned previously the data in Table A08 are not seasonally adjusted, so part of the movement we are seeing can be due to seasonality.
The GSS harmonised definition of “disabled” is determined by two Labour Force Survey (LFS) questions:
whether the health condition or illness lasts longer than 12 months (LNGLST)
whether the condition reduces the respondent’s ability to carry out their day-to-day activities (LIMACT)
In Quarter 3 2017, the increase that we saw in disability was driven mainly by the gateway variable LNGLST while in Quarter 2 2018, the increase was driven mainly by the variable LIMACT.
To put this apparent discontinuity into context, it is useful to look at the longer-term trend in the series (Figure 2). The disability questions on the LFS have undergone changes since 2010 and this has meant that comparisons over time have become difficult to interpret. There are two previous sets of changes that have resulted in discontinuities in the time series:
in January 2010, a rewording of the introduction to the section of the survey covering disabilities
in April 2013, changes to the wording of the disability questions in order to bring the LFS more into line with the definitions and questions used in other household surveys in the UK (see the notes in Dataset A08 for further details)
Consequently, the estimates from 2010 onwards are not directly comparable with those for previous years. Also, the estimates from April 2013 are not comparable with those for either the 2010 to 2012 or the pre-2010 periods. These discontinuities are clearly shown on Figure 2.
From the start of the series (in Quarter 2 1998) up until Quarter 4 (Oct to Dec) 2009, the number of disabled people increased steadily and had begun to level off. Then, between Quarter 4 2009 and Quarter 1 2010, there was a sharp increase in the number of disabled people, the largest on record (912,000), due to the change in wording of the questionnaire. In the subsequent periods, the number of disabled people remained at this higher level until Quarter 2 2013, when there was a large drop of 390,000 due to the definitional changes. Since then, the series had been increasing steadily until the apparent step increase seen in Quarter 3 2017, which was of a similar magnitude to the fall in Quarter 2 2013 and the second-largest quarterly increase (391,000) since the series began.
The difference between the discontinuities in 2010 and 2013 and the apparent discontinuity in 2017 is that the latter does not coincide with any significant change in the questionnaire or interview processes. There have been no changes to the routeing of the survey, the disability questions or any other survey questions in Quarter 3 2017. There have also been no changes to the information provided to respondents prior to the interview. There have been no changes introduced to interviewer training or written guidance. Moreover, all the system processes have been double-checked and everything has run normally. Nothing has been changed in the data rotation specifications either. All this analysis suggests that there is no direct evidence that the survey instrument has caused the change that we saw in Quarter 3 2017.
Moreover, after the discontinuities in January 2010 and in April 2013, the data were following the same trend for a few periods post-discontinuity, while the data since the apparent discontinuity in Quarter 3 2017 appear more volatile.
As can be seen in Figure 3, in Quarter 3 2017 there was an increase in the number of disabled men and women, which was more pronounced for men. In the subsequent two periods (Quarter 4 2017 and Quarter 1 2018) the number of disabled men decreased quite sharply while the decrease for women was more subdued. In Quarter 2 2018, the number of disabled men and women increased but the increase in the number of disabled women was larger. Following this increase, the number of women reporting disability surpassed the level seen in Quarter 3 2017 while the level for men returned to the level seen in Quarter 4 2017.
Figure 4a shows the number of disabled people by age group between Quarter 3 2016 and Quarter 2 2018. Throughout the series, the largest number of disabled people are in the oldest age group (50 to 64-year-olds). There is a clear increase in all age groups between Quarter 2 2017 and Quarter 3 2017 that matched what we saw in the top-level disability estimates.
Indexing the series allows us to see more clearly which age groups were most affected by the apparent discontinuity. In Figure 4b we see that there were increases in all age groups in Quarter 3 2017, with the largest increases in the younger age groups (16 to 17- and 18 to 24-year-olds).
Consistent with the split between the sexes in Figure 3, men have been the main driver for the increase in the 18 to 24-year-olds while the increase in the 16 to 17-year-olds was similar in both sexes (not shown). Following the apparent discontinuity, three age groups (16 to 17, 18 to 24 and 35 to 49 years) have continued their broad downward trend, returning to the trend seen prior to the apparent discontinuity in Quarter 3 2017. The 25 to 34-year-olds and the 50 to 64-year-olds have continued their upward trend. In Quarter 2 2018, the increase in the 25 to 34- and the 50 to 64-year-olds are driven mainly by men while the small increase in the 35 to 49-year-olds is driven entirely by women (not shown).
Figure 5 examines the number of people with disabilities according to their labour market status and highlights that the increases (seen in Figure 1a) were more concentrated in the people classified as being in employment and being economically inactive (those people not in work and either not seeking nor available to work).
For subsequent data periods after the apparent discontinuity in Quarter 3 2017, the employment level of disabled people has followed a broadly upward trend and it has remained elevated while the economically inactive series has been more volatile. Following the apparent discontinuity in Quarter 3 2017, the economic inactivity series decreased for two consecutive periods before a further increase between Quarter 1 2018 and Quarter 2 2018. This discrepancy in the behaviour of people across the different labour market statuses could be because some groups are more likely to change their survey responses to health questions. For example, previous discontinuities have shown that those in employment are more likely to change their reported health status, perhaps because their activities are less severely (or less consistently) limited by their health condition.
The fact that the increase in disability status reporting appears to happen in a very short space of time suggests that it could be either the result of a change in the survey characteristics or a very large and immediate external factor. The remainder of the article will investigate these hypotheses concentrating on the number of disabled people.
Disability reporting across the main survey characteristics
Looking at other aspects of the survey design between Quarter 2 2017 and Quarter 3 2017, there are no clear patterns in the change in the disability data:
there were increases across most response types; however, personal responses saw a bigger increase (in volume terms) than proxy responses (which saw a higher increase in percentage terms) while cases brought forward decreased slightly
there were increases across all regions except Northern Ireland
increases were seen across both modes of interview (face-to-face and telephone), but the increase in face-to-face was more prevalent
increases were seen across most health conditions with legs or feet, depression, bad nerves, learning difficulties, back or neck and other disabilities seeing the largest increases
Reviewing the same survey design aspects between Quarter 1 2018 and Quarter 2 2018, the investigations provide no clear patterns in the change in the disability data:
there were increases across most response types with the exception of proxy responses, with the biggest increase seen in personal responses
there were increases across most regions except the North West, East Midlands, West Midlands and Northern Ireland
increases were again seen across both modes of interview (face-to-face and telephone) but the increase in face-to-face was more prevalent
increases were seen across most health conditions with other disability, depression, bad nerves and mental health seeing the biggest increases
Even though there are some similarities between the two periods, the increases in Quarter 3 2017 were more broad-based.
The role of random fluctuations was examined by looking at the share of the population reporting disability by wave. This enables us to track individual cohorts of respondents over time, as they join the survey at “wave 1” in a given quarter, then are interviewed five times over five consecutive quarters, before leaving after “wave 5”.
Figure 6 shows a comparison between interview waves rather than following the respondents’ journey over the five quarters in which they are surveyed. This allows us to see how the current waves compare historically. In Quarter 3 2017, there was a large increase in the share of respondents reporting a disability status across all waves with the exception of wave 2. In the latest period, Quarter 2 2018, we again see an increase in the disabled population in wave 1, wave 2 and wave 5, with particularly notable increases in waves 1 and 5.
As can be seen from Figure 7, the wave 1 that was interviewed for the first time in July to September 2017 came near the top of the distribution. In subsequent quarters this wave contributed towards the decrease we have seen in the number of disabled people. The wave starting in April to June 2017 also saw decreases in the periods after the apparent discontinuity while the remaining two waves feeding in the July to September period (the waves starting in January to March 2017 and October to December 2016) and staying in the sample for at least one more period had more inconsistent behaviour in subsequent quarters.
Since the apparent discontinuity, we can see that both wave 1s that joined the sample in October to December 2017 and January to March 2018 are towards the middle of the distribution and lower than the wave 1 that joined the sample in July to September 2017. However, the wave 1 that joined the sample in April to June 2018 came at the very top of the distribution. If we follow the respondent journey for wave 5 we can see that it returned to the wave 1 level seen when it joined the sample in April to June 2017. It is not unusual to see an increase between waves 4 and 5; however, wave 5 tends to be lower than wave 1. The remaining three waves feeding into the April to June 2018 period have seen decreases.
The reporting rate for GSS harmonised disabled was tracked across individual survey weeks, to establish whether the change happened at a very specific point in time.
Random fluctuations (and variations in short-term sample structure) make it difficult to pinpoint precise timings, but Figure 8 takes a 13-week moving average of the disability rate over time. It also shows the 13-week moving average of the previous year, for comparison. The timings of recent changes appear to be closely aligned to LFS quarters.
The line increases steadily for around 13 weeks, as the moving average includes more and more Quarter 3 2017 weeks, then appears to reach a more steady state at the end of the quarter, when the moving average is entirely based on Quarter 3 2017 weeks. Extending the chart to Quarter 2 2018 confirms that the Quarter 3 2017 increase broadly levels off at or around the Quarter 3 and Quarter 4 threshold. There is a slight decrease in Quarter 4 2017 and Quarter 1 2018, suggesting there might be a minor “settling down”. In Quarter 2 2018, it looks like the 13-week moving average is returning to the level seen in Quarter 3 2017, still higher than the levels seen one year earlier.
Impact of external drivers on disability reporting status
In addition to examining the LFS, analysis also looked at whether there was any evidence that the step increase in people reporting disability could have been caused by external factors.
External drivers of reporting behaviour are difficult to measure, but we would expect them also to impact on public awareness of disabilities. One proxy measure for public awareness is volumes of Google searches using specific words, with data available from the Google Trends website. Large numbers of searches will not necessarily translate into higher numbers of people reporting disabilities and the timings may be lagged. However, if the discontinuity had been caused by an increase in public awareness, we would expect to see some sort of spike in related Google searches.
Some of the largest search volumes, and largest fluctuations, observed for the period since January 2017 relate to “mental health” – as shown in Figure 9 (indexed to the largest observation on the chart, scaled to 100). The three spikes, in April to May 2017, October 2017 and May 2018, have been annotated with some significant media stories, although we can’t directly measure their effect on public awareness and the list is not intended to be exhaustive. Even though there are spikes in the number of searches for certain words like “mental health” and “stress”, the timing of these spikes doesn’t coincide with the apparent discontinuity in the disability data.
Changes in public awareness may build up cumulatively over a period of time, so it is possible that LFS reporting in July 2017 was influenced by the April to May 2017 spike in public awareness. However, it seems unlikely that there would be a two-month delay before a sudden impact. The most recent spike in May 2018 in Google searches coincides with the increase we have seen in the disability series. This suggests that some of the impact we saw in Quarter 3 2017 could have been influenced by the media events in April and May 2017 (outlined in Figure 9), although it is not clear why this would not trigger an increase in LFS until July 2017.
An assessment was also carried out to determine whether the benefits system had been a driver of reporting changes. As this analysis has concluded, the apparent step-increase in reporting disabilities between Quarter 2 2017 and Quarter 3 2017 was focused largely on people in employment and therefore it was unlikely that benefits administrative data was a direct link to this LFS increase. Administrative statistics confirm that there were no major changes in the trends of numbers of people receiving disability-related benefits at, or shortly before, the July 2017 increase in LFS reporting.Nôl i'r tabl cynnwys
The number of disabled people increased by 391,000 between Quarter 2 (Apr to June) 2017 and Quarter 3 (July to Sept) 2017. Following this apparent discontinuity, the number of disabled people decreased for two consecutive quarters before increasing again between Quarter 1 (Jan to Mar) 2018 and Quarter 2 2018. The increase seen in the latest period (130,000) is not unprecedented and is generally less broad-based than the increase seen between Quarter 2 2017 and Quarter 3 2017.
Analysis of the Labour Force Survey (LFS) suggests that the increases in the number of people reporting disability were more concentrated in the employment and inactivity categories in both Quarter 3 2017 and Quarter 2 2018.
In both Quarter 3 2017 and Quarter 2 2018, there was an increase in the number of disabled men and women. Following the increase in Quarter 2 2018, the number of women reporting disability surpassed the level seen in Quarter 3 2017, while the level for men returned to the one seen in Quarter 4 (Oct to Dec) 2017.
In Quarter 3 2017, there were broad-based increases in disability reporting across all age groups and most waves except wave 2. However, in Quarter 2 2018, there were increases in waves 1, 2 and 5 but the increases in waves 1 and 5 stand out as particularly large. In terms of age groups in Quarter 2 2018, there were increases in people aged 25 to 34, 35 to 49 and 50 to 64 years but only the 25 to 34-year-olds and the 50 to 64-year-olds have seen a large increase and have continued following an upward trend since the apparent discontinuity.
There are some similarities between the survey characteristics impacted by the increases in Quarter 3 2017 and Quarter 2 2018: personal responses have increased, both interview modes were impacted particularly face-to-face and most regions and conditions were also impacted. However, the impact in Quarter 3 2017 was larger and generally more broad-based. Moreover, the increases seen in the health conditions were distributed differently in both periods.
The 13-week moving average shows that the increase that took place in Quarter 3 2017 occurred at the start of the quarter and levelled off to some extent in subsequent periods, before a slight increase in Quarter 2 2018 returned it to the level seen in Quarter 3 2017.
The most recent spike in May 2018 in Google searches coincides with the increase we have seen in the disability series. This suggests that some of the impact we saw in Quarter 3 2017 could have been influenced by the media events in April and May in 2017 (outlined in Figure 9), although it is not clear why this would not trigger an increase in LFS until July 2017.
As a result of the apparent discontinuity and the inconclusive investigations at this stage, comparisons should be treated with caution between Quarter 2 2017 and subsequent quarters. Users should be mindful that the data are not seasonally adjusted so any change that they are seeing between consecutive quarters could be due to seasonality. Furthermore, any short-term changes may be subject to sampling variation; for example, the 95% confidence interval around the number of disabled people in Quarter 2 2018 (7.5 million) is between 7.3 million and 7.6 million.
Users can continue making comparisons prior to the apparent discontinuity, between a given quarter and the same quarter in previous years (we normally advise users to make these “same quarter” comparisons with not seasonally adjusted datasets). Comparisons between periods following the apparent discontinuity should also be treated with caution while investigations continue.
While it will be important to continue monitoring new LFS data as they become available, it seems unlikely that this will enable us to identify a clear cause in the future. However, we will continue to work in consultation with stakeholders such as the Department for Work and Pensions (DWP) and provide an update of this article in November 2018.Nôl i'r tabl cynnwys
Manylion cyswllt ar gyfer y Methodoleg
Ffôn: +44 (0)1633 651599