FOI ref: FOI-2023-1372
You asked
I noted another correction on the Deaths by Vaccination Status output. Please provide the following information:
How do you discover a correction is needed?
What is your Quality assurance procedure?
What is the procedure for corrections?
How many corrections have been published by the team working on this output since the inception of the output?
How many corrections have been published for health, death or birth statistics per year for the last five years?
We said
Thank you for your request. Please see the following answers to your questions:
How do you discover a correction is needed and what is the procedure for corrections?
An error can be identified by internal or external users following publication.
We correct errors in line with the Code of Practice for Statistics by adhering to the following procedure.
When an error is identified, it is the responsibility of the Deputy Director, or delegated authority to judge the error as either minor or major correction. This will consider the impact of the error and the nature of the error. This impact score is calculated by answering the following 3 questions and multiplying the scores from each.
What is your Quality assurance procedure?
The analytical pipeline code produces 2 csv files, one for Age Standardised Mortality Rates (ASMRs) and one for deaths. These spreadsheets show the total number in the population, number of people correctly matched, last vaccination date of 1st, 2nd, 3rd, and 4th dose, earliest of these maximum dates, number of people added from non-National Immunisation Management System (NIMS) data. These are checked against the previous publication for plausibility.
Quality assurance built into the code, checks how many more people were in the population for each vaccination status (0-3) in the new data, compared to the previous version. For 4th doses, we checked if the number was feasible using NHS (National Health Service) data on vaccinations.
These figures are checked against the previous release, checking if the number of deaths in any given month increased (or decreased very slightly), and only to an extent that could be explained by late registrations.
ASMRs from the newest release are compared to the previous release to check those with large changes. For those, we checked if the change was plausible.
Overlapping month periods are compared to previous and new release figures to ensure no anomalous increases in deaths in months where it would be unlikely. Some increases in most recent month are acceptable due to late registrations.
Figures are then presented in graphs and visually inspected for anomalies.
Additional quality assurance code has been implemented to ensure previous mistakes were not repeated such as checking sexes and months were correctly aligned.
How many corrections have been published by the team working on this output since the inception of the output?
The following errors have been corrected by the team working on this output inception.
16 May 2022 - Incorrect age-standardised mortality rates in tables 2 and 5 for the 18-39 age group for the vaccination category 'Second dose at least 21 days ago' (minor error).
21 February 2023 - Months were mislabelled in tables 3 to 5 and the contents labels for tables 3 and 5 listed incorrect months and males and females were mislabelled in tables 3 and 4 (minor error).
25 August 2023 - An issue with some of the data in this release which affected the way person years were summed when incorporating fourth vaccination doses (major error).
How many corrections have been published for health, death or birth statistics per year for the last five years?
Due to changes to our systems, our correction records cover publications from May 2019 onwards. Over the last 5 years, ONS has published approximately more than 3000 bulletins, and a similar volume of releases of a different type (such as articles, datasets, methodological information, etc.). Over this period, the number of corrections made across all releases all releases produced by the Health Analysis and Pandemic Insights Directorate containing the words 'deaths', 'mortality', 'birth', and 'births' are in the associated download named 'Number of corrections made across all releases produced by the Health Analysis and Pandemic Insights directorate containing the words 'deaths', 'mortality', 'birth' and 'births'.
Our Code of Practice for Statistics requires us to publish a correction to any error identified, as soon as practicable, with an explanation and clarity on scale, nature, cause and impact. Quality assurance is undertaken throughout the entire statistical process, to support early identification and correction of any potential errors, as much as it is reasonably possible in line with statistical best practice. In the rare event that an error is recognised in one of our publications, we make every effort to rectify this as quickly as possible. For transparency, we include a blue box notice at the top of our publications to raise awareness of the correction. Errors are actively monitored and review processes are put in place to reduce the chance of reoccurrences.
During the COVID-19 pandemic, our outputs significantly increased to provide rapid and timely data informing the public on public health, as well as the economy and society. Outputs have different frequencies (weekly, monthly, quarterly, annual, and ad-hoc), so it is important to keep this in mind when looking at the correction numbers.
Corrections of publications relating to 'Health'
In order to action your request for the number of corrections to all publications relating to health in general for the last 5 years, we would be required to manually interrogate our records using numerous search terms to ensure that we captured everything in scope of this request, as the topic of 'health' covers a wide range of our publications. The cost limit for FOI is £600, which equates to 24 working hours for 1 person to action, and we believe this would be exceeded to action this request.
We would suggest narrowing the scope of this request so we may answer within the cost limit. We would recommend (as you have done for you Deaths by Vaccinations request) focusing on one particular publication and asking for a specific number of editions of the publication. Please note, the records we hold began in 2019, so we would recommend keeping the time frame between 2019 and 2023.