Cynnwys
1. Introduction
The National Statistics Quality Review (NSQR) of the Living Costs and Food Survey (LCF) was published in May 2016. The review started in January 2014 and reported its findings in May 2016. The review was led by ONS methodologist Jeff Ralph; with independent external expertise provided by Professor Thomas Crossley, Professor of Economics at the University of Essex; and Paul Smith, Associate Professor from the University of Southampton.
The review had a broad scope, assessing the degree to which user requirements are met and whether the data produced is of sufficient quality for the purposes to which it is put. It reviewed various methodological aspects of the survey, including the sample design, data collection instrument, the collection process, coding, imputation and weighting. The review provided a wide range of recommendations, covering both how the current survey can be improved as well as possible alternative data sources to be explored.
This article provides a formal public response to the NSQR, outlining the steps being taken to address the recommendations.
Nôl i'r tabl cynnwys2. ONS response
The overall assessment of the review was that the LCF needs improvement to make it entirely fit for purpose for expenditure estimates; however, it is still the best source of expenditure information for most expenditure categories. The NSQR recommendations outline further work needed to make necessary improvements to the survey, as well as outlining packages of work to explore the potential use of alternative sources of expenditure data.
The NSQR recommendations are categorised into 5 main areas:
- Sample design and estimation
- Data collection
- Processing
- International comparison
- Future developments
Overall there were 30 recommendations, with the following priorities assigned: 12 high; 15 medium; and 3 low. The recommendations will be taken forward in one of 3 ways, through:
- the ONS Data Collection Transformation programme
- establishing a number of research projects within the LCF Research team
- projects led by other experts in ONS, in particular ONS Methodology and Prices
ONS accepts the relevance of all recommendations made. A summary of the actions that will be taken in response to each of the recommendations is outlined in Annex A.
Nôl i'r tabl cynnwys3. Governance
The LCF Steering Group will provide high level oversight of the actions to address the recommendations. To enable them to fulfil this role, more detailed plans to address the recommendations will be presented to the group (these papers are available to other users on request). The group comprises of cross government and external users of LCF data and meets bi-annually.
Progress against the actions outlined will also be monitored by the ONS Quality Centre.
Nôl i'r tabl cynnwys4. Summary
Work has started on a number of recommendations, particularly those being addressed within the Data Collection Transformation Programme as part of the project to harmonise the LCF and the Survey on Living Conditions.
For some projects led by the Research and Methodology teams, funding is being sought to progress recommendations more quickly through commissioning external Research Agencies, for example the project to redesign the existing paper diary.
Therefore, progress with recommendations is dependent on securing resource from outside of the LCF Research team, as well as priorities set within wider ONS initiatives/programmes- both factors will impact on the speed at which progress can be made.
We will continue to keep users informed of the methodological improvements made to the LCF survey via the published methodological notes linked from the Family Spending publication.
Nôl i'r tabl cynnwys5. Annex A
Response to each of the NSQR recommendations
| Recommendation | Priority- High(H); Medium(M); Low(L) | ONS response to recommendation | Status & timescale | |||
| Sample design and estimation | ||||||
| 1. Explore whether a change of design could deliver improved estimates for the same cost. | M | This will be considered as part of ONS's Data Collection Transformation programme, in particular as part of the project to harmonise the LCF and SLC. | Work to start in early 2017 | |||
| 2. Investigate the effect on the precision of levels and change of introducing oversampling areas with high proportions of high income households. | M | This will be considered as part of ONS's Data Collection Transformation programme, in particular as part of the project to harmonise the LCF and SLC. Progress is dependent on access to administrative data and therefore the outcome of the Digital Economy bill. | Work to start in early 2017 | |||
| 3. Carry out further work on the use of LFS and other controls. Include research on the benefits of using the composite calibration framework to improve the quality of LCF estimates on income estimates. | H | Analysis will be conducted by ONS Methodology in conjunction with Southampton University. The project will review the current use of calibration controls from the LFS, exploring options for improvement, leading to recommendations for change and a consistent weighting method for income and expenditure. | In progress- end date March 2017 | |||
| 4. Continue the preliminary analysis of the effect of sampling error in LCF expenditure estimates on price index numbers and publish the results. Extend the analysis to sub-populations. | M | Completed- a paper will be published in the Survey Methodology Bulletin in early 2017. | In progress- end date March 2017 | |||
| Data collection | ||||||
| 5. Carry out a follow-up survey of households included in the LCF sample, both responding and non-responding, to provide further information on non-response bias in the LCF. | M | This will be considered following full integration of the LCF and SLC so that non-response bias is measured on a stable survey instrument. | Earliest start point 2019 | |||
| 6. Consider whether a change in the incentive structure can be used to address low survey participation rates in a cost effective manner. | H | In August 2016 the conditional incentive was increased from £10 to £20, funding is currently available for this until the end of March 2017. Further actions are dependent on the impact of this measure, which will be reviewed in January 2017. | In progress- ongoing | |||
| 7. Review the methodology for allocating field work to geographic areas to ensure a more even distribution within interviewer areas across the year without violating the survey’s design. | M | This is being addressed as part of ONS's Data Collection Transformation programme, in particular as part of the project to harmonise the LCF and SLC. From January 2017 the LCF and SLC sample will be selected jointly, providing a more even workload by interviewer area. | In progress- end date January 2017 | |||
| 8. Review the effectiveness of the new interviewer training materials by carrying out interviewer and interviewer manager focus groups towards the start of the 2016/17 financial year. | M | ONS Field Office, in collaboration with the LCF Research team, are carrying out a review of the LCF training materials, leading to implementation of changes from March 2017. | In progress- end date March 2017 | |||
| 9. Review the quality assurance processes carried out by interviewers by considering whether any of their tasks could be carried out more efficiently by office based staff. | M | A project will be set up to examine the most efficient approach to quality assurance of the paper diary | Not yet started- end date March 2018 | |||
| 10. Carry out further work to investigate the trade-offs between maintaining a 2 week diary period and adopting a shorter diary period. | H | To be addressed as part of the diary re-design project- see recommendation 15. | In progress- end date March 2017 | |||
| 11. Carry out further work to review the level of detail needed for combined payment breakdowns (where bills may not provide the level of detail requested). | M | This recommendation will be taken forward as part of the project set up to develop the LCF questionnaire- see recommendation 14 below. | See rec 14 | |||
| 12. Carry out further analysis on a larger scale to explore the extent of interview length on interview fatigue (including more questionnaire sections asked at different points in the interview and potentially a further survey year to increase robustness of results). | H | This recommendation will be taken forward as part of the project set up to develop the LCF questionnaire- see recommendation 14 below. | See rec 14 | |||
| 13. Carry out further analysis to understand other potential causes of under-reporting in the LCF survey. | H | A project will be set up to carry out quantitative analysis comparing LCF data with other data sources. | Not yet started- end date September 2017 | |||
| 14. Allocate additional resources to the LCF Research Team (including securing funding for DCM resource) to develop and implement a more robust questionnaire and testing process and ensure that the questionnaire design keeps pace with ongoing changes in consumer spending/behavior. | H | The LCF Research team has been restructured to enable a greater focus on questionnaire development. A project will commence in January 2017 to take forward a structured review of expenditure data collection. | Not yet started- ongoing, initial questionnaire changes to be implemented in April 2018 | |||
| 15. Review the adult, child and pocket diaries and update them once every 5 years (to ensure the content remains up to date). The first review (in paper form) should be implemented by April 2017. | H | A project started in December 2016 to review the current diary design leading to recommendations for a revised diary layout/content. Further work will take place during 2017/18 to redesign the diary and carry out further testing prior to implementation in April 2018. | In progress- initial review to report in May 2017 | |||
| 16. Carry out in-depth interviews as a follow-up survey with LCF respondents to understand how respondents go about answering questions of concern, and to assess the accuracy and completeness of diary recording. | H | To be taken forward as part of the questionnaire development project referenced under recommendation 14. Diary aspect to be taken forward as part of outsourced project referred to in recommendation 15 above. | See rec 14 & 15 | |||
| 17. Review the collection of income data within ONS Social Surveys to reduce discrepancies with other sources of income data. | H | This will be addressed as part of ONS's Data Collection Transformation programme, in particular as part of the project to harmonise the LCF and SLC. | In progress- first phase completed in Jan 2017 | |||
| 18. Carry out more work to understand reporting of holiday expenditure. | L | To be taken forward as part of the project to investigate causes of under reporting (rec 13) and the diary redesign project (rec 15) | See rec 13 & 15 | |||
| 19. Consider web diary collection for the LCF alongside other ONS web collection initiatives. | H | This will be considered as part of ONS's Data Collection Transformation programme. | Not yet started- timescale TBC | |||
| Processing | ||||||
| 20. Submit the LCF edit strategy and the general principles underlying the imputation strategy into the development of the ONS Business Process Model for Social Surveys. | L | The planned review of editing and imputation as part of the project to harmonise the LCF and SLC will apply the principles embedded in the current LCF approach where appropriate. Therefore ensuring best practice from the current LCF approach is applied more widely. ONS Methodology will support this activity. | Not yet started- end date December 2017 | |||
| 21. Review and document how editing and imputation are carried out, paying particular attention to the efficacy of the diary imputation. | L | The review of editing and imputation of data collected within the Face to Face interview will be taken forward as part of ONS's Transformation programme, in particular as part of the project to harmonise the LCF and SLC. Review of diary imputation is dependent on securing resources outside of the LCF team. | Not yet started- end date December 2017 | |||
| 22. Discuss potential benefits from and access to supermarket product information with market research companies and supermarkets. | M | Acquisition of commercial data, including scanner data from retailers, is being taken forward by the Data Access team within Admin Data Division. Findings will be considered as part of the Transformation Programme. | In progress- ongoing | |||
| 23. Explore the possibility of semi-automated coding of purchase information from scanned supermarket receipts. | M | A project has been set up to explore the development of automated coding. Subsequent actions will depend on the outcome of the interim project, including potential to implement till receipt scanning. | In progress- project to report in March 2017 | |||
| International comparison | ||||||
| 24. Liaise with other countries and organisations undertaking web data collection, receipt scanning, and automatic coding to understand more about the data quality obtained. If the quality is satisfactory, undertake experiments to evaluate these approaches in the LCF. | M | Liaison with other countries and organisations is established as good practice within ONS. Existing documentation and papers will be reviewed in collaboration with the ONS Prices Development team. The findings from this review will be used to inform the work plan to address recommendation 19. | In progress- ongoing | |||
| 25. Engage with ONS programmes to enhance the use of administrative data on income, though it should be acknowledged that record-level linkage is likely to be a long-term aspiration. | M | A Non-Survey Data (NSD) team has been established in ONS Social Surveys to work closely with the Administrative Data Division and Data Collection Transformation Programme. Engagement and requirements are well established. A Key Dependency is the Digital Economy Bill passing into legislation. | In progress- ongoing | |||
| 26. Maintain links and promote information sharing with other NSIs and organisations (particularly the Understanding Society team at Essex) with respect to the development of expenditure surveys. In particular, progress by the US and Norwegian statistical offices should be monitored. | M | Liaison with other countries and organisations is established as good practice within ONS. | In progress- ongoing | |||
| Future developments | ||||||
| 27. Explore the differences between market research data and LCF expenditure data, comparing expenditures at multiple levels of disaggregation. Evaluate the accuracy of panel data constrained to higher level LCF expenditure values. | M | ONS will provide a funding contribution to sponsor a PhD student in the Business School at the University of Huddersfield (from September 2017) to carry out the analysis required; the work will be supported by ONS Methodology. The work will establish the tools, techniques and processes to enable expenditures to be estimated from the panel data. | Not yet started- first interim report to be completed March 2018 | |||
| 28. Investigate the practicality of a market research company running a specific panel to provide ONS with additional expenditure information to supplement LCF data. | M | Next steps to be confirmed following the outcome of recommendation 27. | Not yet started- dependent on findings from recommendation 27 | |||
| 29. Continue working to secure access to store scanner data. | H | Acquisition of commercial data, including scanner data from retailers, is being taken forward by the Data Access team within Admin Data Division. Findings will be considered as part of the Transformation Programme. | In progress- ongoing | |||
| 30. Review international use of store scanner data and plan a programme of research to determine the best future use of this data. | H | A project has been set up in conjunction with Cardiff Business school to address this. | In progress- to be completed June 2017 | |||
| Source: Office for National Statistics | ||||||