Progress Report in response to OSR Assessment of LCF, March 2022

Last updated:
3 October 2022

Background

In July 2021, the Office for Statistics Regulation (OSR) published a report assessing the Living Cost and Food Survey (LCF) for compliance with the Code of Practice for Statistics.

This report identified two requirements and three considerations for improvements to be made to ensure that the LCF and its outputs continue to be fit for purpose and for Family Spending in the UK to retain its National Statistics status.

This document presents updates on our work to meet the OSR requirements, as of March 2022, and planned future actions.

Requirement 1

Demonstrate a positive direction of travel by making some short-term gains by the end of 2021 as follows:

  1. The Office for National Statistics (ONS) needs to take remedial action to improve the stability of the existing LCF processing system and to develop a new system, which meets the needs of users and the staff running the systems.
  2. The ONS should enhance its understanding of the value of the statistics by improving its engagement with users, within and outside ONS, to capture a wide range of views and use these to drive its priorities for development. ONS should reflect on the Government Statistical Service's User Engagement Strategy for Statistics to help determine the best methods for engaging with users.
  3. The ONS should provide a mechanism and relevant access for other teams in ONS who make use of LCF data to be able to contribute to the quality assurance of the data.

Actions taken as of March 2022

The ONS had set up a dedicated project, the LCF and RPI improvement project, that has delivered:

  • detailed process maps, covering the current LCF data processing and identifying the "pain points"

  • recommendations for improving the processes and their governance

  • commitment to the full rebuilding of the LCF data processing system module by module; the rebuild will be complete by March 2023, but single modules are being deployed one by one into operations throughout the course of the development

  • the enhanced Data Quality Assurance Module in R and the new Data Aggregation Module in Python are being implemented in the Business as Usual (BAU) operations of the LCF Research team; the formal handover is scheduled for 28 March 2022

The LCF and RPI improvement project was closed on 16 February 2022, with the work on the full rebuilding of the LCF data processing system, improvement to the business processes and governance of Questionnaire Changes moving to the newly established Household Financial Statistics Transformation (HFST) Project.

The LCF Development team, established as part of the LCF and RPI improvement project to oversee the review of the LCF Downstream methods and processes, has been budgeted until the end of 2025 fiscal year, rebranded to Household Financial Survey (HFS) Pipeline Development, and tasked to develop and maintain the uplifted LCF data processing system, and the statistical processing pipelines of the other Household Financial Surveys, applying the principles of Reproducible Analytical Pipelines.

The LCF Research team, responsible for the LCF outputs, continues to host regular Curiosity meetings with the internal users of the data. In those meetings the team shares their findings about and interpretations of the LCF data, giving chance for data users outside the LCF research team to contribute to the quality assurance of the data. These meetings have been well attended and highly appreciated.

The LCF Research team also provides access to the micro level unweighted and weighted data, alongside the aggregated data to the Economic Social and Environmental Group (ESEG) data users.

The Curiosity meetings and the ESEG users' access to LCF data is proving greatly beneficial to ensuring the quality of the LCF data by providing more structured quality assurance process, enabling conversations, and sharing different perspectives on the data.

The LCF Steering group is the forum where the LCF Research team discusses the progress and performance of the survey with the internal and external stakeholders. The Steering Group is the vehicle for collecting, reflecting, and prioritising the user needs. Recently the Steering Group increased the frequency of its meetings (from one, or two times a year to quarterly) in order to invigorate the interaction between survey delivery and the stakeholders. The LCF Research team has also initiated discussions with the existing LCF steering group members to refresh its terms of reference, membership list, and purpose as part of the strengthening of our engagement across the LCF user community.

Next steps

  • Continue improving and updating the legacy modules of the LCF processing system. During the years, LCF data processing systems have organically grown, with additions of subroutines, derivations, and modules that have outlasted their use. The complete system re-development aims at streamlining the data processing, pruning un-necessary code, and improving code transparency by aligning it with the best coding practice in developing reproducible analytical pipelines. Though the system re-development is mostly tactical, it follows the ONS policy to move away from the use of SAS and strives to exploit opportunities for using strategic tools, and existing frameworks and standards to facilitate a future strategic uplift. Timeline: by the end of March 2023.

  • Developing and delivering training for the LCF Research team to upskill them in the use of the updated modules of the LCF processing system. Timeline: ongoing, as the various pipeline modules are implemented in business-as-usual activities.

  • Review the governance around the LCF and its development, both internally and externally, and consider different channels of engagement to capture a wider range of views and uses. This will need to consider the wider Household Financial Statistics Transformation project, as part of which wider engagement plans are being developed. Timeline: continuous.

Requirement 2

Publish a plan, which includes specific actions, deliverables, and a timetable by the end of March 2022, that explains how it will address the following strategic improvements:

  1. The ONS needs to develop a solution to address user need for more granular breakdowns of data, so that the devolved administrations and other key users can use the statistics in the ways that they need to for the public good.

  2. The ONS needs to invest time and resource in pursuing initiatives to improve the quality and robustness of LCF data. The ONS should be open to creative solutions to improve the response rate, such as continuing to explore the use of different short and long-form questionnaires/diary, alternative sampling strategies and linking with other data sources, rather than focusing only on increasing the existing sample.

Actions taken as of March 2022

The Household Financial Statistics Transformation (HFST) Project has been initiated with the aim to exploit the full power of alternative data sources and establish integrated, efficient, inclusive survey operations, to provide higher quality, more timely and more granular statistics, which meet the key user needs for household financial statistics and unlock an in-depth understanding of income, wealth, spending and financial resilience across the UK population. A new Deputy Director has been appointed to lead the project.

The ONS had appointed and commissioned NatCen to conduct an external review aimed at informing the ONS statistical blueprint for its future expenditure data collection, including a high-level survey design. The project included:

  • a user requirement exercise to identify what household expenditure data our users need and for what purpose (Work Package 1)

  • an assessment of alternative, non-survey data that may be available to meet part of these needs (Work Package 2)

  • recommendations for the best design for a household expenditure survey to meet user needs, including quality and cost requirements (Work Package 3)

The review officially commenced at the beginning of November and concluded at the end of March 2022. The recommendations of the review will be taken forward and implemented through the HFST Project. The progress on the NatCen review achieved by March is as follows:

  • Work package 1: A report has been produced, summarising the requirements of the interviewed household expenditure data users.

  • Work package 2: Interviews for the review of alternative data sources and a final report reviewed by the ONS has been completed. This report contains an overview of alternative sources of data for Household expenditure detailing availability/maturity of data, as well as an assessment of suitability against the user needs highlighted in work package 1.

  • Work package 3: A final report outlining recommendations for a high-level design of expenditure data collection has been produced and will be shared both internally within the ONS and via the LCF steering group. NatCen's review agreed with the OSR assessment that a survey will be needed for the foreseeable future, but recommended the ONS continue to explore and exploit administrative data to augment the survey collection. In the shorter term, anything that can be done to automate the collection, coding and processing will have the largest beneficial impact on the survey. Moving to digital capture for the diary element of the survey would be the most significant change the survey could make. Many other countries are moving towards this and developing web-based solutions for data capture.

A project for testing expenditure data collection via a mobile app, developed by Statistics Netherlands as an alternative to the current LCF diary, was set up and trialled. An internal one-week trial of the app consisting of approximately 20 volunteers was run in February 2022. The aim was to obtain feedback on the usability and functionality of the English offline version of the app as it is.

While there were limitations to the usability testing, the feedback from volunteers provided beneficial insight into aspects of the benefit and functionality of this version of the app. The test confirmed known areas and identified others in need of adaptation for a UK setting and for a potential quantitative pilot with LCF respondents. There are several improvements and possible adaptations that could be beneficial to consider before any other testing is carried out. Options for building an in-house hosting platform for the app and/or development of a browser-based solution are also being explored.

Next steps

Consideration 1

The ONS should consider extending the scope of its LCF and RPI improvement project work to include input from some of its important external users, such as those in the devolved administrations, where additional intelligence could be gathered on the use and issues faced by the government in its use of LCF data.

Actions taken as of March 2022

As part of the LCF external review by NatCen, a detailed user requirement exercise has been carried out to inform the future of its household expenditure data collection. A report has been produced, summarising the requirements of the interviewed LCF data users.

Next steps

Please see the next steps to address requirements 1 and 2 above.

Consideration 2

The ONS should consider the management of risks throughout the end-to-end production process as part of the LCF projects medium-term work and ambitions.

Actions taken as of March 2022

The ONS has expanded the scope of the LCF and RPI improvement project to consider wider aspects of the end-to-end process. We have established a continuous improvement project aiming at streamlining the LCF receipts processing and data coding and editing processes.

Work on producing an LCF Quality Assurance of Administrative Data (QAAD) has started. The plan is to be ready with it by the end of April 2022.

The ONS has initiated the development of robust questionnaire change control process. It evolved to include the harmonisation of the Household Financial Surveys' questionnaire change control processes. So far, this workstream investigated the current questionnaire change control processes used in each of the Household Finance Surveys, gathered user needs, developed to-be journeys, and designed the proposals for the future governance of the questionnaire change processes. A comprehensive Change Control Check list was developed to provide a standard of good practice and user guide.

Next steps

  • Complete Quality Assurance of LCF data report using the Quality Assurance of Administrative Data framework at A3 level of the QA matrix, in order to fully assess the end-to-end risks to the survey. Timeline: By the end of April 2022.

  • Finalise the Change Control governance structure, defining the roles and responsibilities; finalise the user requirements and choose the optimal technical solution for automated change control system as part of the HFST Project.

Consideration 3

The ONS should determine a longer-term solution for the LCF which draws on a broader base of data, international best practice, and wider transformation initiatives.

Actions taken as of March 2022

Please see the actions taken and next steps to address Requirement 2. In addition, the ONS is continuing its collaboration and exchange of knowledge with Australian Bureau of Statistics and restarting the collaboration with other NSIs (for example, Statistics Canada and Statistics Netherlands). Timeline: Continuous.