In November 2017 Office for National Statistics (ONS) suspended Dataset A08 (Labour market status of disabled people) for subsequent time periods due to an apparent discontinuity between Quarter 2 (Apr to June) 2017 and Quarter 3 (July to Sept) 2017.
We investigated the potential reasons for the change. While the evidence presented in this article is not conclusive we cannot identify any quality issues and we have reinstated the dataset from 15 May 2018.
The apparent step increase in the number of people reporting disabilities1 seen in Quarter 3 (July to Sept) 2017 Labour Force Survey (LFS) has decreased to some extent in the subsequent periods for the harmonized standard definition of disabled people2. However, the number of people reporting disability remains elevated. The comparisons in this article have been done on not seasonally adjusted data so some of the change could be attributed to seasonal variation.
Analysis of the LFS design did not identify any cause that could have introduced a change in the pattern of disability reporting; increases were seen across most response types and all of the interview modes.
Analysis of the LFS suggests that the increases in the number of disabled people were more concentrated in the employment and inactivity categories in Quarter 3 (July to Sept) 2017. There were increases across all age bands and in the proportion of respondents reporting disability across all waves except wave 2.
Even though the step change in the number of disabled people has decreased to some extent in the subsequent periods we will require further data points to determine whether this increase can be attributed to sample variation.
As a result of the apparent discontinuity and the inconclusive investigations at this stage comparisons should be treated with caution between Quarter 2 (April to June) 2017 and subsequent quarters. Users should be mindful that the data is not seasonally adjusted so any change that they are seeing between consecutive quarters could be due to seasonality.
Users can continue making comparisons prior to the apparent discontinuity, between a given quarter and the same quarter in previous years (ONS normally advise users to make these ‘same quarter’ comparisons with not seasonally adjusted datasets).
Comparisons between periods following the apparent discontinuity should also be treated with caution while investigations continue. Further recommendations on ‘same quarter’ comparisons after the discontinuity will be given in November when we publish the data for Quarter 3 (July to September) 2018 and we have a whole year of data after the apparent discontinuity.
Notes for: Main points
- In the Labour Force Survey (LFS) respondents self-identify themselves as disabled or not disabled.
- The Government Statistical Service (GSS) Harmonised Standards focus on a “core” definition of people whose condition currently limits their activity. In summary the core definition covers people who report:
(current) physical or mental health condition(s) or illnesses lasting or expected to last 12 months or more
the condition(s) or illness(es) reduce their ability to carry out day-to-day activities
We suspended publication of disability employment figures, based on the Labour Force Survey (LFS) and the Annual Population Survey (APS), due to an apparent discontinuity between Quarter 2 (Apr to June) 2017 and Quarter 3 (July to Sept) 2017. Working with stakeholders such as the Department for Work and Pensions (DWP), we conducted a review of the datasets to determine why the step change happened. We were concerned that it might have resulted from quality problems. While the analysis did not result in a conclusive reason being identified, this article presents the analysis that has been carried out before reinstating the dataset and describes the action plan we will lead on to provide further recommendations for comparability of the data historically.Nôl i'r tabl cynnwys
There has been a steady increase in the UK population aged 16 to 64 over the last few years, averaging around 150,000 more people per year.Figure 1a shows an increase of almost 400,000 in the number of people aged 16 to 64 reporting a disability under the harmonised standard definition (disabled people) between Quarter 2 (Apr to June) 2017 and Quarter 3 (July to Sept) 2017. Over the same period, there was a fall of 363,000 in the number of people aged 16 to 64 who are not disabled according to the harmonised standard (either because they have stated that they are not disabled or their condition doesn’t meet the Government Statistical Service (GSS) harmonised definition) (not disabled) (Figure 1b). Following the apparent discontinuity between Quarter 2 2017 and Quarter 3 2017 the number of disabled people decreased by around 60,000 per quarter while the number of not disabled people has seen a slightly faster increase.
Figure 1: Number of people, ages 16 to 64 (not seasonally adjusted), between April to June 2013 and January to March 2018.
As can be seen in Figure 2 in Quarter 3 2017 there was an increase in the number of disabled men and women which was more pronounced for men. In the subsequent periods the number of disabled men decreased quite sharply while the decrease for women was more subdued.
Figure 3 examines the number of people with disabilities according to their labour market status and highlights that the increases (seen in Figure 1a) were more concentrated in the people classified as being in employment and being economically inactive (those people not in work and either not seeking or available to work). For subsequent data periods employment level of disabled people has remained elevated while the economically inactive level for this group has returned to the trend consistent with the data points seen before the apparent discontinuity in Quarter 3 2017. This discrepancy in the behaviour of people across the different labour market statuses could be because some groups are more likely to change their survey responses to health questions. For example, previous discontinuities have shown that those in employment are more likely to change their reported health status, perhaps because their activities are less severely (or less consistently) limited by their health condition.
The fact that the increase in disability status reporting appears to happen in a very short space of time suggest that it could be either the result of a change in the survey instrument, survey characteristics or a very large and immediate external factor. The remainder of the article will investigate these three hypotheses concentrating on the number of disabled people.
Disability reporting across key survey characteristics
Looking at other aspects of the survey design between Quarter 2 2017 and Quarter 3 2017, there are no clear patterns in the change in the disability data:
there were increases across most response types; however, proxy responses saw a bigger increase than personal responses while cases brought forward decreased slightly
there were increases across all regions except Northern Ireland
increases were seen across both modes of interview: face-to-face and telephone
increases were seen across most health conditions
The role of random fluctuations was examined by looking at the share of the population reporting disability by wave. This enables us to track individual cohorts of respondents over time, as they join the survey at “wave 1” in a given quarter, then are interviewed five times over five consecutive quarters, before leaving after “wave 5”.
Figure 4 shows a comparison between interview waves rather than following the respondents’ journey over the five quarters in which they are surveyed. This allows us to see how the current waves compare historically. In Quarter 3 (July to Sept) 2017 there was a large increase in the share of respondents reporting a disability status across all waves with the exception of wave 2.
As can be seen from Figure 5 the wave 1 that was interviewed for the first time in July to September 2017 came at the top of the distribution. In subsequent quarters the wave contributed towards the decrease we have seen in the number of disabled people. The wave starting in April to June 2017 also saw decreases in the periods after the apparent discontinuity while the remaining two waves feeding in the July to September period (the waves starting in January to March 2017 and October to December 2016) and staying in the sample for at least one more period had more inconsistent behaviour in subsequent quarters.
Since the apparent discontinuity we can see that both wave 1s that joined the sample in October to December 2017 and January to March 2018 are towards the middle of the distribution and lower than the wave 1 that joined the sample in July to September 2017.
When looking at disability by age (Figure 6) there was an increase across all age groups in Quarter 3 2017. It is usual to have an increase across all age groups at the same time. Moreover, the biggest increases in percentage terms took place among the younger age groups (16- to 17-year-olds and 18- to 24-year-olds). Following the apparent discontinuity there have been decreases on the quarter across all age groups with the exception of the 25- to 34-year-olds who are still increasing but at a slower pace and the 50- to 64-year-olds who saw an increase in Quarter 4 (Oct to Dec) 2017.
Analysis of the survey component
We have looked at the survey to determine whether a change in the survey could have caused the movement in Quarter 3 2017 similar to the discontinuities which took place in 2010 and 2013. The investigation confirmed that there have been no changes made to the disability related questions. There have been no changes to the interviewer instructions or processes. This meant that there is no direct evidence that the survey instrument has caused the change that we saw in Quarter 3 2017.
Impact of external drivers on disability reporting status
In addition to examining the Labour Force Survey (LFS), analysis also looked at whether there was any evidence that the step increase in people reporting disability could have been caused by external factors.
External drivers of reporting behaviour are difficult to measure, but we would expect them also to impact on public awareness of disabilities. One proxy measure for public awareness is volumes of Google searches using specific words, with data available from the Google Trends website. Large numbers of searches will not necessarily translate into higher numbers of people reporting disabilities, and the timings may be lagged. However, if the discontinuity had been caused by an increase in public awareness, we would expect to see some sort of spike in related Google searches.
Some of the largest search volumes, and largest fluctuations, observed for the period since January 2017 relate to “mental health” – as shown in Figure 7 (indexed to the largest observation on the chart, scaled to 100). The two spikes, in April to May and October 2017, have been annotated with some significant media stories, although we can’t directly measure their effect on public awareness and the list is not intended to be exhaustive. Even though there are spikes in the number of searches for certain words like mental health and stress the timing of these spikes doesn’t coincide with the apparent discontinuity in the disability data. Changes in public awareness may build up cumulatively over a period of time, so it is possible that LFS reporting in July 2017 was influenced by the April to May spike in public awareness. However, it seems very unlikely that there would be a two-month delay before a sudden impact.
An assessment was also carried out to determine whether the benefits system had been a driver of reporting changes. As the analysis above has concluded, the apparent step-increase in reporting disabilities was largely focused on people in employment and therefore unlikely that benefits administrative data was a direct link to this LFS increase. Administrative statistics confirm that there were no major changes in the trends of numbers of people receiving disability-related benefits at, or shortly before, the July 2017 increase in LFS reporting.Nôl i'r tabl cynnwys
Analysis of the Labour Force Survey (LFS) suggests that the number of people reporting disability increases were more concentrated in the employment and inactivity categories in Quarter 3 (July to Sept) 2017. The large increase in LFS respondents reporting a disability was across all age bands and across all waves except wave 2. There were increases across most response types and across all regions except Northern Ireland. The increases were also seen across both modes of interview and across most health conditions.
Even though this apparent step change has decreased to some extent in the subsequent periods we will require further data points to determine whether this peak can be attributed to sample variation.
As a result of the apparent discontinuity and the inconclusive investigations at this stage comparisons should be treated with caution between Quarter 2 (April to June) 2017 and subsequent quarters. Users should be mindful that the data is not seasonally adjusted so any change that they are seeing between consecutive quarters could be due to seasonality. Users can continue making comparisons prior to the apparent discontinuity, between a given quarter and the same quarter in previous years (ONS normally advise users to make these ‘same quarter’ comparisons with not seasonally adjusted datasets). Comparisons between periods following the apparent discontinuity should also be treated with caution while investigations continue. Further recommendations on ‘same quarter’ comparisons after the discontinuity will be given in November when we publish the data for Quarter 3 (July to September) 2018 and we have a whole year of data after the apparent discontinuity.
While it will be important to continue monitoring new LFS data as they become available, it seems unlikely that this will enable us to identify a clear cause in the future. However, we will continue to work in consultation with stakeholders such as Department for Work and Pensions (DWP) and provide updates of this paper in August and November 2018.Nôl i'r tabl cynnwys
Manylion cyswllt ar gyfer y Methodoleg
Ffôn: +44 (0)1633 651599