FOI REF: FOI-2023-1370
You asked
Can you please provide the following in relation to the Deaths by Vaccination Status output:
How do ONS discover a correction is needed, and what is the procedure?
What is their Quality assurance procedure?
How many corrections have been published by the team working on this output since the inception of the output?
How many corrections have been published for mortality statistics per year for the last five years?
We said
Thank you for your request. Please see the following answers to your questions.
How do ONS discover a correction is needed, and what is the procedure?
An error can be identified by internal or external users following publication.
We correct errors in line with the Code of Practice for Statistics by adhering to the following procedure.
When an error is identified, it is the responsibility of the Deputy Director, or delegated authority to judge the error as either minor or major correction. This will consider the impact of the error and the nature of the error. This impact score is calculated by answering the following 3 questions and multiplying the scores from each:
Q1. Does the error impact upon the key points/messages within the release and therefore likely to lead to misinterpretation of the statistics?
No, this is a notification for a routine change
No, key points/messages unaffected.
Yes, key points affected but key trends/messages unchanged - likelihood of misinterpretation low.
Yes, key points/messages affected and therefore likelihood of misinterpretation.
Q2. What is the relative size of the error?
The error is relatively small and is within stated levels of accuracy (e.g. 95% confidence interval.
Yes, key points/messages affected and therefore likelihood of misinterpretation.
Q3. Does the error feed into other inputs?
No
Yes
The definition decides whether the correction is either major or minor.
The questions take account of a range of considerations, such as:
The impact of the error (will it lead users to misinterpret the statistics? does it affect other outputs?)
The nature of the error (is it the headline number or a minor figure in a historical table that is incorrect?)
Errors with scores of 10 and over are categorised as major and those with scores of 9 and under are categorised as minor. In the instance of the latest Deaths by Vaccination Status output, the error was categorised as major.
A correction notice is then published alongside the affected publication to inform users of the error while work to correct the error begins.
The corrected information is then loaded into a correction template and sent for divisional director approval.
The content is loaded to our preview website by our publication officers and checked by the analyst responsible. Once these checks have been made and quality has been assured, the corrected content will be published, superseding the original content and the UK Statistics Authority will receive a link to both the correction template and corrected content.
What is the ONS Quality assurance procedure?
a. The analytical pipeline code produces 2 csv files, one for Age Standardised Mortality Rates (ASMRs) and one for deaths. These spreadsheets show the total number in the population, number of people correctly matched, last vaccination date of 1st, 2nd, 3rd, and 4th dose, earliest of these maximum dates, number of people added from non-National Immunisation Management System (NIMS) data. These are checked against the previous publication for plausibility.
b. Quality assurance built into the code, checks how many more people were in the population for each vaccination status (0-3) in the new data, compared to the previous version. For 4th doses, we checked if the number was feasible using NHS (National Health Service) data on vaccinations.
c. These figures are checked against the previous release, checking if the number of deaths in any given month increased (or decreased very slightly), and only to an extent that could be explained by late registrations.
d. ASMRs from the newest release are compared to the previous release to check those with large changes. For those, we checked if the change was plausible.
e. Overlapping month periods are compared to previous and new release figures to ensure no anomalous increases in deaths in months where it would be unlikely. Some increases in most recent month are acceptable due to late registrations.
f. Figures are then presented in graphs and visually inspected for anomalies.
g. Additional quality assurance code has been implemented to ensure previous mistakes were not repeated such as checking sexes and months were correctly aligned.
How many corrections have been published by the team working on this output since the inception of the output?
The following errors have been corrected by the team working on this output inception.
16 May 2022 - Incorrect age-standardised mortality rates in tables 2 and 5 for the 18-39 age group for the vaccination category 'Second dose at least 21 days ago' (minor error).
21 February 2023 - Months were mislabelled in tables 3 to 5 and the contents labels for tables 3 and 5 listed incorrect months and males and females were mislabelled in tables 3 and 4 (minor error).
25 August 2023 - An issue with some of the data in this release which affected the way person years were summed when incorporating fourth vaccination doses (major error).
How many corrections have been published for mortality statistics per year for the last five years?
Due to changes to our systems, our correction records cover publications from May 2019 onwards. Over the last 5 years, ONS has published approximately more than 3000 bulletins, and a similar volume of releases of a different type (such as articles, datasets, methodological information, etc.). Over this period, the number of corrections made across all releases produced by the Health Analysis and Pandemic Insights Directorate containing the words 'deaths' and 'mortality' are shown in the associated download named 'Number of corrections made across all releases produced by the Health Analysis and Pandemic Insights Directorate containing the words 'deaths' and 'mortality'.
Our Code of Practice for Statistics requires us to publish a correction to any error identified, as soon as practicable, with an explanation and clarity on scale, nature, cause and impact. Quality assurance is undertaken throughout the entire statistical process, to support early identification and correction of any potential errors, as much as it is reasonably possible in line with statistical best practice. In the rare event that an error is recognised in one of our publications, we make every effort to rectify this as quickly as possible. For transparency, we include a blue box notice at the top of our publications to raise awareness of the correction. Errors are actively monitored and review processes are put in place to reduce the chance of reoccurrences.
During the COVID-19 pandemic, our outputs significantly increased to provide rapid and timely data informing the public on public health, as well as the economy and society. Outputs have different frequencies (weekly, monthly, quarterly, annual, and ad-hoc), so it is important to keep this in mind when looking at the correction numbers.