1. Background

The UK Labour Force Survey (LFS) National Statistics Quality Review (NSQR) was published in February 2014. The review focused largely on implications for accuracy arising from the LFS design, fieldwork and estimation procedures. In addition, it outlined that increasing requests for single month estimates have called into question the timeliness of results. The main conclusion from the review was that the LFS enables the production of good quality estimates from the survey outputs. However, 27 specific recommendations were put forward to improve the LFS design and procedures.

Nôl i'r tabl cynnwys

2. Progress

An Office for National Statistics (ONS) response to the National Statistics Quality Review (NSQR) was published in March 2016, outlining how the high level recommendations would be addressed. Of the 27 recommendations, 13 are in progress or ongoing, seven are complete or and a further seven have been closed. These cover a range of topics, a number of which are relating to continuous improvement of the questionnaire and Labour Force Survey (LFS) quality. Several recommendations are being taken forward using resource from the Office for National Statistics (ONS) Social Surveys Transformation programme, including investigation of a rolling data collection period, moving reference weeks and electronic data collection. Other projects are being taken forward within the LFS research team or via projects led by other experts at ONS.

As a result of work to address NSQR recommendations, a number of methodological papers have been published:

Progress against each of the recommendations is provided in Section 4.

Nôl i'r tabl cynnwys

3. Governance

The Labour Force Survey (LFS) Management Board, which meets quarterly and has coordinated response to the National Statistics Quality Review (NSQR) to date, provides high-level oversight of the actions to address the recommendations. Since the NSQR was conducted in 2014, many of the recommendations of the review have been addressed to improve the design, collection and processing of LFS data. While not all recommendations have been implemented on the LFS, any further development will be wrapped up in the development of the mixed-mode online first Labour Market Survey (LMS), which is intended to replace the LFS in the future. The NSQR recommendations are reflected in the user requirements for the LMS, which will be scrutinised as part of the National Statistics Accreditation process for the labour market statistics using the LMS data.

Nôl i'r tabl cynnwys

4. Labour Force Survey National Statistics Quality Review response

The following table lists each of the 27 recommendations from the 2014 NSQR and their progress. Details of each recommendation including the status in 2016 and progress updates in 2019 and 2021 are included in the following sections.

Recommendation 1: Allocate dedicated resource to continuous quality improvement on the Labour Force Survey

ONS response to recommendation, March 2016

Examples of resource allocated to continuous quality improvement are as follows.

From a response rate perspective: a Local Area Response Project Team has been set up to research fieldwork patterns and develop action plans, in interview areas where response rates have fallen most since 2011.

From a questionnaire perspective: Office for National Statistics (ONS) has implemented workshops aiming to identify questions for removal, such that questionnaire focus can be kept relevant whilst not increasing its length. Further, ONS has worked with the Department for Business, Innovation and Skills (BIS) to initiate a review of Labour Force Survey (LFS) education questions.

From the perspective of providing data to meet user needs: ONS will conduct an LFS reweighting process each year, to use the most up-to-date population estimates and projections. Further, ONS has put in place regular production of longitudinal datasets based on Annual Population Survey (APS) data, to supplement LFS longitudinal data. Data from Eurostat ad hoc modules is now available to government users on the main APS files.

From the perspective of support for data users: in 2015, in addition to providing updated LFS User guides 2, 3, 4 and 6, resource was allocated to publishing a new User guide 10 'Analysis of data collected by the Labour Force Survey'.

Update on progress, February 2019

This work is ongoing. Further examples of continuous quality improvement work between 2016 and 2019 are given in this section.

Response rates: a restinting project is under way and due to be completed in 2019 (see recommendation 5). The Behavioural Insights Team has carried out some investigatory work to identify some behavioural barriers to response and a number of experiments are being taken forward, such as telephone scripting and reminder texts. The concept of the Messenger Effect on advance materials is also being tested from November 2018.

Questionnaire: dedicated resource has been added to the Questionnaire Development Team. A review of the education section of the survey has been instigated to improve quality and reduce respondent burden, and a piece of work is under way to look at the collection of nationality data with the Northern Ireland Statistics and Research Agency (NISRA).

Providing data to meet user needs: the LFS reweighting process has been reviewed to use the latest mid-year population estimates and subnational population projects, producing reweighted datasets every other year at the minimum. Three-year pooled APS datasets have been produced for users from August 2017, to be released on an annual basis.

Support for data users: User guide 9 'Eurostat' was updated in 2018. Separate User guide 9a created to provide information relating to EU-LFS ad hoc modules. This volume is updated annually to coincide with the publication of ad hoc module data on the APS. An update of User guide 5 'Classifications' will be published in spring 2019.

Update on progress, July 2021

In March 2020, as a result of the coronavirus (COVID-19) pandemic, the LFS had to change the way it contacted people for initial interviews, from face-to-face interviewing to telephone-based. This change in method of initial contact has had an impact on both the level of response and the non-response bias of the survey, and consequently the survey estimates. A number of measures were put in place to ensure the quality of the LFS data is maintained or improved were possible.

Data collection: Previously all interviewing for Wave 1 and some for follow-up waves were conducted face-to-face, except for addresses North of the Caledonian Canal, which were conducted over the phone from Wave 1 onwards. With the onset of the pandemic all interviews were switched to telephone interviewing. To implement this change, the sample was tele-matched and an online portal was set-up where respondents were invited in the advance letter to provide telephone numbers. All interviewers were equipped with additional materials to carry out telephone interviewing from their own home. Further information can be found in the Labour Force Survey performance and quality monitoring reports

Meeting user needs: As LFS Wave 1 rates halved with the onset of the pandemic and the switch to telephone interviewing, the issued sample size was doubled from July 2020 to March 2021. While the achieved sample experienced a drop in the period April to June 2020, the sample boost from July resulted in an achieved sample size similar to pre-pandemic levels. From April 2021 the sample boost has been reduced from 100% to 60% alongside the roll out of a new field activity called Knock to Nudge. This involves interviewers visiting addresses where no phone contact details are available to try and get these at the doorstep.

The change in non-response bias was significantly evident in a change to the housing tenure of the Household Reference Person, with a lower proportion of rented addresses being included and an increase in the proportion of those owned outright by the occupier. To mitigate the impact of this non-response bias, we have introduced housing tenure into the LFS weighting methodology. Work to further refine weights to account for an under-representation on the non-UK born population is ongoing. A reweighting exercise will be conducted in the second half of 2021.

Response rates: Just before the onset of the pandemic, a text messaging services was implemented as a keep-in-touch-exercise. The process involves a short reminder text message being sent to respondents who provide a mobile phone contact at the previous wave. The texts indicate to the respondent that an interviewer will be in touch to conduct the next Wave interview. A previous trial carried out on the LFS by the ONS Behavioural Insights team over the course of 2018 and 2019 assessing the messenger effect (using male or female signatories of different seniority for advance letters, for example) on response rates showed that the current signatory provided the highest level of response rates. Therefore, no changes were made to the advance materials. The ONS Behavioural Insight team also conducted a staff motivational experiment using email messages to boost motivation and productivity. The findings from the experiment were inconclusive because of the small sample size and were not taken further.

Status, July 2021

Ongoing

Recommendation 2: Ensure LFS representation and liaison with ONS projects whose outcomes have major implications for design of LFS

ONS response to recommendation, March 2016

ONS is represented on Eurostat task forces, for example, those considering the proposed introduction of household income questions and the 2018 ad hoc module. For the former task force, research was conducted by ONS recommending that household income not be included in LFS; this recommendation was implemented by Eurostat.

ONS also participates in Eurostat's biannual Labour Market Statistics Workshops and annual European LFS Methodology Conference, and will host the latter in 2016. Of particular note is Eurostat's draft regulation IESS (Integrated European Social Statistics), targeted for introduction in 2019: ONS is actively planning and communicating necessary activities to comply with IESS. It will participate in IESS Implementation Task Forces in 2016 to 2017.

Substantial liaison has taken place with ONS's Electronic Data Collection Programme, and going forward with the Social Statistics Transformation Programme, focusing on implementing online data collection and utilising administrative and big data.

Update on progress, February 2019

ONS has had continued involvement in Eurostat task forces and working groups and will continue to engage on an international level.

ONS Social Statistics Transformation, which covers projects to develop the question content and processes, are represented on the LFS Steering Group and Management Boards, and Social Surveys representatives also sit on the equivalent Transformational Boards, both at an operational and senior level.

Update on progress, July 2021

The ONS Social Survey Operations has had continued involvement in the ongoing development of the online-first Labour Market Survey (LMS) and the wider Integrated Population and Characteristics Survey (IPACS), and the ONS Social Survey Transformation continues to feed into the LFS Steering Group.

The ONS Social Survey Operations worked with business areas across the ONS to develop measures to address the impact of the pandemic on the way survey work was delivered. The ONS is also represented in a working group involving the Market Research Society (MRS), the Social Research Association, other data collection agencies and Heads of Professions from across government with the aim to share our plans to return to in-home interviewing in alignment to MRS and GSR guidance as well as government restrictions.

Status, July 2021

Ongoing

Recommendation 3: Retain the Postcode Address File for now, but move to AddressBase in future

ONS response to recommendation, March 2016

The relevance and benefit of the Postcode Address File and AddressBase is being considered by the ONS Address Register project, which has a cross-government focus. LFS, in line with other ONS surveys, will utilise address register findings to define future sample frame methodology.

Update on progress, February 2019

The Social Statistics Transformation Programme, which will deliver a mixed-mode, administrative data-first approach to labour market statistics, will use AddressBase as its primary sampling frame. The move to this frame for the LFS and other social surveys is still being considered.

Update on progress, July 2021 There are no immediate plans to change the LFS sampling frame from the Postcode Address File to AddressBase. The online-first LFS uses AddressBase as the primary sampling frame.

Status, July 2021

Closed

Recommendation 4: Discontinue sampling of NHS accommodation

The ONS response to recommendation, March 2016

A project will address this work package. It has not been given highest priority to date given the small number of NHS addresses in the LFS sample.

Update on progress, February 2019

Not yet progressed due to priorities of other work.

Update on progress, July 2021

The ONS Social Survey Operations is conducting an impact assessment of potential options.

Status, July 2021

In progress

Recommendation 5: Explore the efficiency of interviewer stint areas, with a view to redesigning

The ONS response to recommendation, March 2016

Analysis has suggested that the current interviewer areas are inefficient, as numerous fieldwork demands have changed since the areas were defined. As a result, ONS is conducting a project to implement a new restint design, in 2016 or 2017. It is anticipated that future restinting exercises will be required, driven by developments in online interviewing and use of administrative data.

Update on progress, February 2019

The stint areas are being redesigned as part of a project, which is due to be implemented in 2019.

Update on progress, July 2021

Because of the UK leaving the European Union, the regulations (including precision requirements) for the Integrated European Social Statistics (IESS) are not applicable for the ONS's LFS outputs. This means that no increase to the sample size for face-to-face interviewers needed to be made. Given the change in data collection with the onset of the pandemic, the complex nature of the changes that would be required when restinted areas are implemented, the concerns around the possible impact on the system, and the implications for the allocation and management of work by the interviewers it was deemed appropriate to pause this work.

Status, July 2021

Closed

Recommendation 6: Explore potential for adaptive fieldwork to focus resources more efficiently

ONS response to recommendation, March 2016

A modelling approach has been developed (drawing on advice from ONS Methodology and from Southampton University) relating to cases entering week two of data collection in Telephone Operations. Domains of cases are identified and a propensity model applied to prioritise cases more efficiently. A test of the approach will be implemented in 2016.

Update on progress, February 2019

An adaptive fieldwork design trial has been run in the Telephone Unit from August 2018. Results will be evaluated in spring 2019.

Update on progress, July 2021

The findings from the research trial run in 2018 highlighted that the responsive fieldwork design was successful. The design used response behaviour in the previous Wave and the first week of fieldwork for the current wave to calculate response likelihood in the second week. This was then used to prioritise cases for the remainder of the fieldwork period. The trial showed that response rates have been maintained while caseloads could be covered more efficiently. The responsive fieldwork design has been implemented for LFS telephone interviews with August 2019. Further testing is required before this design can be implemented for boost cases.

Status, July 2021

In progress

Recommendation 7: Further reduce the number of wave 1 telephone interviews

ONS response to recommendation, March 2016

The proportion of wave 1 cases handled by Telephone Operations was halved in 2014, to approximately 10 to 15%. It is estimated that this helped improve wave 1 response rates by 1 to 2%.

Update on progress, February 2019

The proportion of telephone interviews in wave 1 has reduced further from January 2018 in order to further improve response. Only wave 1 cases north of the Caledonian Canal are now collected by telephone, around 1 to 2% of the Great Britain sample. Impact analysis is being carried out to determine if this had a significant effect.

Status, February 2019

Complete

Recommendation 8: Pursue international agreement on a more standardised approach to reporting response and for greater clarity in Eurostat publications about differences in how response is measured and reported**

ONS response to recommendation, March 2016

A Eurostat Task Force presented recommendations that were proposed for testing by some national statistical institutes (NSIs) in 2015. The results of the testing are awaited. ONS will review outcomes and consider the best way forward, taking into account harmonisation with other surveys.

Update on progress, February 2019

Not progressed due to priorities of other work.

Update on progress, July 2021

Because of UK exiting the European Union with the end of 2020, the ONS is liaising with other international organisations (e.g. OECD) to establish requirements for standardised reporting.

Status, February 2019

In progress

Recommendation 9: Explore ways of exploiting the option to have a longer data collection period

ONS response to recommendation, March 2016

This is being considered as part of ONS's Social Statistics Transformation Programme, in particular online interviewing development.

Update on progress, February 2019

This work is being considered as part of ONS's Social Statistics Transformation Programme. A longer, rolling data collection period is being used as part of a mixed-mode statistical test being carried out from October 2018.

Update on progress, July 2021

The ONS Social Survey Transformation ran a trial in 2019 testing among other things a mixed mode data collection approach, with online first over two weeks, and a further six weeks for a multi-mode data collection period including telephone and face-to-face interviews. Further development is ongoing to conclude on the optimum data collection period using a mixed mode approach.**Status, July 2021**\ In progress

Recommendation 10: Explore the option of using a moving reference week, in the context of the Electronic Data Collection Project

ONS response to recommendation, March 2016

This will be considered as part of ONS's Social Statistics Transformation Programme, in particular online interviewing development.

Update on progress, February 2019

See recommendation 9.

Update on progress, July 2021

The ONS Social Survey Transformation ran a trial in 2019 testing the impact of a rolling reference week on estimates. Initial results indicated that the difference in estimates would be marginal. The ONS is looking to implement a rolling reference period into the Labour Market Survey data collection process with the start of the mixed mode data collection. The impact of using a rolling reference week will then be evaluated on a larger scale.

Status, February 2019

Complete

Recommendation 18: Investigate opportunities for re-weighting the longitudinal flow estimates

ONS response to recommendation, March 2016

A paper was published with findings, in the Survey Methodology Bulletin, autumn 2015, which concluded that the current methodology is appropriate and will be retained.

Following this review, more detailed flows statistics have begun to be published on a quarterly basis. These include estimates of job to job flows and flows between different occupational groups.

Update on progress, February 2019

The first recommendation from the autumn 2015 report was to consider the replacement of the current economic activity calibration variable with that of a lower breakdown level. This recommendation is currently being reviewed.

Update on progress, July 2021

The previously made recommendation to replace the current economic activity calibration variable with that of a lower breakdown level has been reviewed and implemented between the end of 2019 and beginning 2020.

Status, July 2021

Complete

Recommendation 19: Continue to produce rolling quarterly estimates as the preferred monthly headline indicator

ONS response to recommendation, March 2016

In place, unchanged.

Update on progress, February 2019

Complete

Status, February 2019

Complete

Recommendation 20: Apply minor modifications to the seasonal adjustment of single month estimates

ONS response to recommendation, March 2016

Completed - series have been reviewed and seasonal adjustment has been improved. Wave data were previously using rolled forward seasonal factors but are now fully seasonally adjusted every month.

Update on progress, February 2019

Complete

Status, February 2019

Complete

Recommendation 21: Continue with current single month estimates benchmarking method

ONS response to recommendation, March 2016

In place - continuing as usual until the state space modelling approach is introduced.

Update on progress, February 2019

Complete

Status, February 2019

Complete

Recommendation 22: Commence regular production of estimates of sampling variability at earliest opportunity

ONS response to recommendation, March 2016

Not to be developed given progress against recommendation 24.

Update on progress, February 2019

Not being actively progressed.

Status, February 2019

Closed

Recommendation 23: Improve metadata for single month estimates

ONS response to recommendation, March 2016

From January 2016, wave analysis has been published alongside single month estimates with clearer series descriptions.

Update on progress, February 2019

Complete

Status, February 2019

Complete

Recommendation 24: Continue the existing project to develop state space models, but with higher priority

ONS response to recommendation, March 2016

A prototype state space model has been developed that produces single month estimates of unemployment, employment and inactivity from quarterly LFS microdata. In short, the estimates represent time series with the identifiable survey design effects removed.

The methodology has undergone initial peer review and is sufficiently developed to enable the monthly publication of an experimental time series. The plan is to publish these Experimental Statistics alongside the existing LFS single month estimates from mid-2016 onwards.

In the longer-term it is anticipated that the modelled estimates will replace the existing estimates and will be provided for use in the Eurostat monthly unemployment release. The quarterly LFS aggregates will remain as the headline National Statistics.

Update on progress, February 2019

ONS continues working on the prototype state space model. Additional work is required before the estimates can be published alongside the current single month estimates.

Update on progress, July 2021

The new state space modelled estimates are now being published alongside the old version of the single month estimates. The intention was to replace the old single month estimates with these new ones. However, because of a strong user need, it has been decided to publish both versions in parallel.

An article was published in 2019 detailing the new methodology. This change has been implemented using LFS data. Further work is ongoing looking at APS data.

Status, July 2021

In progress

Recommendation 25: Developments of the communal establishment sampling frame from Beyond 2011 are considered for their potential for sampling purposes in the LFS

ONS response to recommendation, March 2016

This is a longer-term recommendation of the National Statistics Quality Review. In the interim the estimated impact of not surveying communal establishments in the LFS was reviewed as part of the project that reconciles LFS estimates of jobs with those produced by ONS business surveys. The review took account of information from the 2011 Census of Population and a methodological article (PDF, 134KB) was published.

Update on progress, February 2019

This is not being actively progressed at this point in time. The recommendation may be taken forward using AddressBase as a sampling frame.

Update on progress, July 2021

Not being actively progressed. The ONS Social Survey Transformation is considering the sampling of communal establishments as part of the wider development of the Integrated Population and Characteristics Survey (IPACS).

Status, July 2021

Closed

Recommendation 26: When available, the potential of improving the weighting using administrative data be examined

ONS response to recommendation, March 2016

This will be considered as part of Social Statistics Transformation.

Update on progress, February 2019

This work has not yet started as the ONS does not yet have access to suitable administrative data.

Update on progress, July 2021

Not being actively progressed. The ONS Social Survey Transformation is considering the use of administrative data in the weighting process as part of the wider development of the Integrated Population and Characteristics Survey (IPACS).

Status, July 2021

Closed

Recommendation 27: Introduce web data collection when systems and design issues are resolved

ONS response to recommendation, March 2016

This will be considered as part of Social Statistics Transformation.

Update on progress, February 2019

Development of an online data collection system is continuing as part of the Social Statistics Transformation work. Initial testing in 2017 established that a good initial response rate (about 25 to 30%) can be achieved to an online first survey invite.

A statistical test is being carried out from October 2018 to establish what mixed mode response rate can be achieved with an online first approach, and how this compares with the current LFS. At the same time, this test will compare estimates from the new survey with comparable estimates in the current LFS. In addition, there will be early tests of new data collection systems.

Update on progress, February 2019

The ONS has published reports from the statistical test and attrition test, which focused on online data collection only including three longitudinal waves.

The launch of the online only LMS was brought forward as a response to the pandemic and went live as a large-scale online data collection exercise in 2020. The collected data are considered experimental and internally compared to monthly LFS estimates.

In February 2021, the ONS conducted a telephone mode test with the aim to assess the impact on response rates. Result from the telephone mode test were positive and led to a planned roll-out in July 2021.

The wider Integrated Population and Characteristics Survey (IPACS) design is being developed with user requirements around web data collection in mind.

Status, July 2021

In progress

Nôl i'r tabl cynnwys

Manylion cyswllt ar gyfer y Methodoleg

Debbie Curtis
socialsurveys@ons.gov.uk
Ffôn: +44 (0)1633 45 5678