Coronavirus (COVID-19) Infection Survey quality report: August 2022

• Overall, in the initial pilot stage of remote data collection, around 17 out of 20 people invited opted in to take part.

• There was no clear pattern during the initial pilot stage for when participants were likely to complete the questionnaire in their 14-day testing window.

• Around 9 out of 10 participants who completed the feedback survey during the initial pilot stage were "satisfied" or "very satisfied" with the new data collection method.

In the week ending 21 July 2022:

• point estimates of the percentage of people testing positive for coronavirus (COVID-19) produced by remote data collection were within the credible intervals of the point estimates produced by study worker home visit collection for England, Wales, Northern Ireland, Scotland, and all English regions except for the East of England

• point estimates of the percentage of people testing positive for COVID-19 produced by remote data collection were within the credible intervals of the point estimates produced by study worker home visit data collection for most age groups

• There were minimal differences between estimates produced by remote data collection methods compared with estimates produced by study worker home visit data collection. We will start presenting estimates for combined data collection methods in our weekly Coronavirus (COVID-19) Infection Survey bulletin on 19 August 2022

The coronavirus (COVID-19) pandemic has had a profound impact across the UK. In response to the pandemic, the COVID-19 Infection Survey (CIS) measures:

• how many people across England, Wales, Northern Ireland, and Scotland would have tested positive for a COVID-19 infection, regardless of whether they report experiencing symptoms

• the number of people who would have tested positive for antibodies against SARS-CoV-2 at different levels

Since the start of the survey in April 2020, questionnaire data and swab and blood samples have been collected by study worker home visits to participants. From July 2022 we have changed the way that we collect our data, moving from study worker home visits to a more flexible remote data collection approach. We have introduced a digitalised questionnaire which participants can complete online or by telephone as well as sending swab and blood sample kits through the post or by courier (for some participants).

This is the first report presenting analysis of the effects of data collection methods. These analyses compare data collected during the initial pilot stage between study worker home visits in the majority of participants, and our new remote data collection approach. This quality report covers information on remote uptake, completion rates and user satisfaction. It also includes comparisons between the different data collection methods on our headline estimates of the number of people testing positive for COVID-19 by country, region, and age. This report presents current opt-in rates during the main remote collection stage. We are planning future analysis to continue to investigate any further observations of this data collection method change, which will be published over the coming months.

The positivity rate is the percentage of people who would have tested positive for COVID-19 on a polymerase chain reaction (PCR) test at a point in time. We use current COVID-19 infections to mean testing positive for SARS-CoV-2, with or without having symptoms, on a swab taken from the nose and throat. Data are based on confirmed positive COVID-19 test results from those living in private households, excluding those living in care homes or other communal establishments.

Data presented in this report are also provided in the Coronavirus (COVID-19) Infection Survey quality report: August 2022 dataset.

We assessed the quality of the data in the Coronavirus (COVID-19) Infection Survey (CIS) in our CIS methods and further information article, which provides information on the data and methods used when study worker visits were undertaken for all participants. The CIS Quality and Methodology Information (QMI) article details the quality characteristics, strengths and limitations, existing users, and suitable uses of the data and methods used. The most recent updates of these reports were for the period up until 30 June 2022, prior to the move from study worker home visit to remote data collection.

This quality report presents comparisons of the data collected during the move to remote data collection, whereby data were collected via both the original method of study worker home visits and by remote methods. A number of analyses were undertaken on these data to compare data collection by each different method, and to investigate if the move to remote data collection affected responses.

In July 2022, we started our new data collection approach where questionnaires are completed by participants either online or by telephone, and swab and blood samples are returned by post or (for a small number of participants), by courier. Each month, participants are given a 14-day "testing window" to complete the questionnaire and return their test sample(s).

Before this change, we invited a small number of existing Coronavirus (COVID-19) Infection Survey (CIS) participants to test the new ways of collecting data, swabs and blood so that we could identify any issues and make adjustments before rolling out the new method to the rest of our participants. This initial pilot stage was purposely designed to test whether invitations sent, and questionnaires completed, online or by telephone would be a successful approach and provide accurate data.

There were two sampling phases in this initial pilot stage. Just over 2,000 participants were selected in phase one with the purpose of making sure the new method of data collection worked for a wide range of user needs. They were selectively invited based on criteria that would test the capabilities of the new online platform across the breadth of participants for example, age range, technological skills, and mobility issues. Therefore, the participant sample used in phase one was not representative of the UK population. The first invites for phase one were sent on 28 March 2022.

In phase two, a stratified sampling methodology was used to select around 2,000 participants that represented the wider survey population. The purpose of phase two was to understand how participants engaged with the new method of data collection on the initial digitalised platform and so this sample was more representative of the UK population. The first invites for phase two were sent on 12 May 2022.

Participants were asked to opt into continuing their participation in the survey using remote data collection. The option to opt in online only became available in phase two where it was anticipated to encourage a greater opt-in rate. When opting in, participants answered questions around how they would like to complete their questionnaire and return their test kits.

Once the initial pilot was completed in July 2022 and when the survey fully moved over to remote data collection, those who had participated in the initial pilot automatically continued providing data and samples. They did not need to opt in again.

The following subsections focus on remote data collection opt-in rates, questionnaire completion rates, and user satisfaction of participants in the initial pilot stage only.

Remote data collection opt-in rate

The remote data collection opt-in rate is defined as the number of participants who accepted the invitation to take part in (opt into) the new remote data collection approach for CIS. The figures in this section relate to the initial pilot stage only.

Overall, in the initial pilot stage, around 17 out of 20 people invited opted in to take part (85.8%).

Up to 24 July 2022 in phase one, 2,414 invites were sent with an overall opt-in rate of 88.3%, and in phase two 2,387 invites were sent with an overall opt-in rate of 83.2%. In both phase one and phase two, around two in five participants opted in through the remote approach in the first five days of receiving their invite letter (40.8% and 42.3% respectively). For both phase one and phase two, uptake started to slow down after one month.

In phase two, reminders were sent 30 days after the invite which resulted in an additional 165 participants opting in. In phase one, reminders were sent out at different points in time in a staggered approach, so the pattern of opting in was not clear. This suggests that reminders did encourage participants to opt into the new remote data collection approach and were more effective when all sent together rather than in smaller batches.

Figure 1 presents the opt-in rates for both phases of the initial pilot stage. The step increases in the opt-in rates around day 50 in phase one and around day 30 in phase two are attributed to participants responding to reminders received.

Remote opt-in rate by number of days after the start of each phase are presented in the Coronavirus (COVID-19) Infection Survey quality report: August 2022 dataset.

Figure 1: Opt-in rate for phase one and phase two of the initial pilot stage of the new remote data collection approach

UK, 28 March to 24 July 2022

Download the data

.xlsx

Questionnaire completion rates

Completion rate is defined as the number of participants who actually completed the questionnaire, either online or over the phone, in the 14-day "testing window" that participants are given to complete their questionnaire. This section outlines questionnaire completion rates only, although participants should also take and return their swab samples in the 14-day window. Sample completion rates will be included in a future publication.

In phase one, 1,467 participants opted into the remote data collection approach and did not opt out of their first 14-day testing window. Testing windows started at different times depending on when participants opted into the remote data collection approach. The first testing windows started on 19 May 2022. Of all participants in phase one, nine out of ten completed a questionnaire (91.1%).

In phase two, 1,980 participants opted into the remote data collection approach and did not opt out of their first 14-day testing window. The first testing window started on 25 May 2022. Of all participants in phase two, 17 out of 20 completed a questionnaire (86.8%).

This gives an overall completion rate for the questionnaire in the first testing window of the Initial pilot stage as 88.7%.

In both phases, there was no clear pattern for when participants were likely to complete the questionnaire in their 14-day testing window.

Questionnaire completion rates will be monitored over future testing windows. Data will also be monitored to identify any factors that affect the rate of questionnaire completion.

Questionnaire completion rates for phase one and phase two of the initial pilot stage are presented in the Coronavirus (COVID-19) Infection Survey quality report: August 2022 dataset.

User satisfaction

Participants in the initial pilot stage were given the opportunity to provide feedback at the end of the questionnaire through a survey. Feedback could only be collected from participants who successfully reached the end of the questionnaire. As part of this survey, participants were asked "How would you rate your overall experience of taking part in the COVID-19 Infection Survey with the new approach?". They were given five options ranging from "Very unsatisfactory" to "Very satisfactory" to select from. For those doing the questionnaire over the phone, the contact centre staff let a proportion of participants know they could contact an Office for National Statistics (ONS) research team to provide feedback. In these cases, the research team read the survey questions and response options over the phone, selecting the correct response for the participant.

This feedback survey was available across the entire initial pilot stage for participants to access at any time, up until 4 July 2022. So, it cannot be determined if participants who responded to this survey were from phase one or phase two of the initial pilot. The satisfaction score does not include responses from participants who are part of the wider roll-out of this new approach.

Overall, around 9 out of 10 participants (89.3%) who completed the feedback survey were satisfied or very satisfied with the new service. For those who were unsatisfied with the new approach, issues with the approach have been identified and improvements are on-going. Most responses came from participants who completed the questionnaire online.

Participant satisfaction responses for the initial pilot stage are presented in the Coronavirus (COVID-19) Infection Survey quality report: August 2022 dataset.

Following feedback from the initial pilot, the online platform was upgraded in the main remote collection stage of moving participants from study worker home visits to remote data collection, with significant improvements in performance and usability. The move from study worker home visit to remote data collection and swab and blood sample returns by post or by courier began in July 2022. The final study worker home visits occurred on 31 July 2022, and all participants will have been invited to move to the remote data collection approach by the end of August 2022.

Participants who had not been involved in the initial pilot stage were invited to opt into the new remote approach week by week in 10 cohorts using a staggered approach throughout July and August 2022, so during this time data were collected by study worker or remote methods. Figure 2 shows the number of questionnaires completed by both collection methods.

The data in this section are also provided in the Coronavirus (COVID-19) Infection Survey quality report: August 2022 dataset.

Figure 2: Study worker visits and questionnaires completed remotely since the start of the move to remote data collection

UK, 6 June to 7 August 2022

Download the data

.xlsx

Opt-in rates for each country during the main remote collection stage up to 11 August 2022 are presented in Table 1. Opt-in rates for cohorts 9 and 10 will be presented in our next quality report.

To assess the impact of the change in how the data were collected on our modelled estimates, we compared two different sets of results. The first set consisted of data and swabs collected by study worker home visits only, and the second set consisted of data collected remotely and swabs returned through the post or courier. The analyses in this section compare these two sets of data.

Included in the comparisons were eight weeks of data for England, Wales, Northern Ireland, and Scotland up to 21 July 2022. Data for the second set of results were only available for the most recent 1.5 weeks analysed, from 11 to 21 July 2022. To enable our models to run, the first 6.5 weeks uses data from study worker home visits. The comparisons for the most recent period provide a reliable comparison between the two different data collection methods. The reference date for the most recent week was 18 July 2022. This is a different reference week to that published in the Coronavirus (COVID-19) Infection Survey bulletin on 5 August 2022 for study worker home visit data collection only so that these comparisons use exactly the same time periods and include the greatest amount of data from the remote data collection method.

The data used for this comparison may include some data from the initial pilot stage for participants whose second test window may not have closed until after 10 July 2022.

The data in this section are also provided in the Coronavirus (COVID-19) Infection Survey quality report: August 2022 dataset, along with comparisons of sample counts and 2-week and 6-week underlying numbers.

Estimated percentage testing positive for COVID-19

The comparison of modelled estimates of the percentage of people testing positive for COVID-19 by data collection method is presented in Figure 3. The most recent week analysed was 15 to 21 July 2022, with a reference day of 18 July 2022. The point estimates produced for the most recent week analysed by remote data collection were within the credible intervals of the point estimates produced by study worker home visit data collection. Comparisons of the estimated percentage of people testing positive for COVID-19 were:

• for England, 4.71% (95% credible interval: 4.49% to 4.92%) by study worker home visit data collection, and 4.92% (95% credible interval: 4.69% to 5.18%) by remote data collection

• for Wales, 4.61% (95% credible interval: 3.71% to 5.62%) by study worker home visit data collection, and 5.20% (95% credible interval: 4.23% to 6.28%) by remote data collection

• for Northern Ireland, 6.06% (95% credible interval: 4.59% to 7.65%) by study worker home visit data collection, and 4.78% (95% credible interval: 3.71% to 5.98%) by remote data collection

• for Scotland, 5.22% (95% credible interval: 4.30% to 6.20%) by study worker home visit data collection, and 4.93% (95% credible interval: 4.14% to 5.83%) by remote data collection

Trends either matched or were broadly similar between study worker home visit and remote data collection. In the week ending 21 July 2022, the percentage of people testing positive for COVID-19 decreased in England, Wales, and Scotland through both remote and study worker data collection methods. In the same week in Northern Ireland, the trend in the percentage testing positive was uncertain for study worker home visit data collection, but the percentage decreased for remote data collection. In the final week in Northern Ireland there were more data collected remotely which provided more information that week than study worker home visit collected data alone which is likely to explain the change in trend. We are therefore confident in the overall similarity in trend.

Figure 3: The percentage of people testing positive for coronavirus (COVID-19) by data collection method

Modelled daily percentage of the population testing positive for COVID-19 on nose and throat swabs, by data collection method, UK, 27 May to 21 July 2022

Notes:

1. Because of smaller sample sizes, there is a higher degree of uncertainty in our estimates for Wales, Northern Ireland and Scotland, compared with England and these results are based on smaller sample sizes as we move to remote data collection. This is shown by wider credible intervals. 

2. This comparison includes data from 3 June to 21 July 2022 for England, and from 27 May to 21 July 2022 for Wales, Northern Ireland and Scotland.

3. Remote plus study worker home visit data collection includes one and a half weeks of the most recent data collected by remote methods (main remote collection) from 11 to 21 July 2022 plus study worker home visit data for the rest of the time series analysed to be able to run the model.

Download the data

.xlsx

The point estimates of the percentage of people testing positive for COVID-19 produced for the week ending 21 July 2022 by remote data collection were within the credible intervals of the point estimates produced by study worker home visit data collection for all English regions, except for the East of England. In the East of England, the estimate by remote data collection was higher, and slightly outside of the credible intervals, produced by study worker home visit data collection. However, the credible intervals around the estimates for the East of England from both methods overlapped.

In the week ending 21 July 2022, the trends in the percentage of people testing positive for COVID-19 in regions of England also either matched or were broadly similar for study worker home visit data collection and remote data collection.

Figure 4: The percentage of people testing positive for coronavirus (COVID-19) by region of England and data collection method

Modelled daily percentage of the population testing positive for COVID-19 on nose and throat swabs, by region and data collection method, England, 3 June to 21 July 2022

Notes:

1. Because of smaller sample sizes, there is a higher degree of uncertainty in our estimates for English regions compared with England overall, and these results are based on smaller sample sizes as we move to remote data collection. This is shown by wider credible intervals.

2. Remote plus study worker home visit data collection includes one and a half weeks of the most recent data collected by remote methods (main remote collection) from 11 to 21 July 2022 plus study worker home visit data for the rest of the time series analysed to be able to run the model.

Download the data

.xlsx

Additional quality assurance checks were conducted on the estimated percentage of people testing positive for COVID-19 by data collection method. Estimates from the remote data collection passed on all checks. Additional quality assurance checks included analysis of:

• Root Mean Square Error (RMSE), which refers to the level of unexplained variance, for example, the variance the model cannot account for; we consider a RMSE value below 0.8 an acceptable threshold

• Rhat, which refers to the measure of the model convergence, for example, how well converged the Bayesian model is; we consider a Rhat value below 1.05 an acceptable threshold

Grouped age over time

In the week ending 21 July 2022, the point estimates of the percentage of people testing positive for COVID-19 produced by remote data collection were within the credible intervals of the point estimates produced by study worker home visit data collection for most age groups.

For age groups school Year 7 to school Year 11, aged 35 to 49 years, and aged 70 years and over, estimates produced by remote data collection were slightly outside the study worker home visit data collection credible intervals. The credible intervals around the estimates from both methods overlapped except for those in school Year 7 to school Year 11, but we are confident in the similarity of estimates between the two methods. The one- and two-week trends for those in school Year 7 to school Year 11 and the two-week trend for those aged 70 years and over were also slightly different between the two methods, reflecting this uncertainty. All other trends were broadly similar with each other.

Figure 5: The percentage of people testing positive for coronavirus (COVID-19) by grouped age and data collection method

Modelled daily percentage of the population testing positive for COVID-19 on nose and throat swabs by age group and data collection method, England, 27 May to 21 July 2022

Notes:

1. Because of smaller sample sizes, there is a higher degree of uncertainty in our estimates for each age group in England compared with England overall, and these results are based on smaller sample sizes as we move to remote data collection. This is shown by wider credible intervals.

2. Remote plus study worker home visit data collection includes one and a half weeks of the most recent data collected by remote methods (main remote collection) from 11 to 21 July 2022 plus study worker home visit data for the rest of the time series analysed to be able to run the model.

Download the data

.xlsx

Single year of age over time

As a result of smaller sample sizes as we move to our new remote data collection method, single year of age analysis for Northern Ireland is not available.

In the week ending 21 July 2022, the trends in the percentage of people testing positive for COVID-19 by single year of age were very similar by both data collection methods for England, Wales, and Scotland.

Modelled estimates by single year of age for England, Wales, and Scotland are provided in the Coronavirus (COVID-19) Infection Survey quality report: August 2022 dataset.

Additional quality assurance checks were conducted on the estimated percentage of people testing positive for COVID-19 over time by single year of age and data collection method. Estimates from the remote data collection passed on all checks. Additional quality assurance checks included analysis of:

• RMSE, as explained previously in this section

• R-squared, which refers to how much variance in the data the model can account for, for example, a measure of goodness of fit

We also checked the deviance, a very similar measure to R-squared but more suitable for non-Gaussian models, to check the assumption that deviance would be the same or similar across runs. Deviance values produced by both collection methods were within 5 units.

Based on the quality findings presented in this report, we are confident that there are minimal differences between samples collected by remote data collection methods compared with data collected by study worker home visit. We will therefore be combining data collected from these two methods for time points where both are in use in future publications. The benefit of combining the data is to achieve the maximum possible sample, to increase the power in our analysis and enable us to resume publishing estimates sooner. We will start presenting estimates for combined data collection methods in our weekly Coronavirus (COVID-19) Infection Survey bulletin on 19 August 2022.

We are also undertaking further comparative analyses which we will present in additional Coronavirus (COVID-19) Infection Survey quality reports over the coming months. These reports will include comparisons on:

• odds ratios from analysis on the impact of the different data collection methods on various outcome measures

• representativeness of participants providing data by remote collection over time

• antibody positivity estimates from blood samples

The Coronavirus (COVID-19) Infection Survey analysis was produced by the Office for National Statistics (ONS) in collaboration with our research partners at the University of Oxford, the University of Manchester, UK Health Security Agency (UK HSA) and Wellcome Trust. Of particular note are:

• Sarah Walker - University of Oxford, Nuffield Department for Medicine: Professor of Medical Statistics and Epidemiology and Study Chief Investigator
• Koen Pouwels - University of Oxford, Health Economics Research Centre, Nuffield Department of Population Health: Senior Researcher in Biostatistics and Health Economics
• Thomas House - University of Manchester, Department of Mathematics: Reader in Mathematical Statistics
• Anna Seale - University of Warwick, Warwick Medical School: Professor of Public Health; UK Health Security Agency, Data, Analytics and Surveillance: Scientific Advisor

Age groups for children and young people

• "Aged 2 years to school Year 6" includes children in primary school and below
• "school Year 7 to school Year 11" includes children in secondary school
• "school Year 12 to those aged 24 years" includes young adults who may be in further or higher education

Those aged 11 to 12 years and those aged 16 to 17 years have been split between different age categories depending on whether their birthday is before or after 1 September.

Confidence interval

A confidence interval gives an indication of the degree of uncertainty of an estimate, showing the precision of a sample estimate. The 95% confidence intervals are calculated so that if we repeated the study many times, 95% of the time the true unknown value would lie between the lower and upper confidence limits. A wider interval indicates more uncertainty in the estimate. Overlapping confidence intervals indicate that there may not be a true difference between two estimates. For more information, see our methodology page on statistical uncertainty.

Credible interval

A credible interval gives an indication of the uncertainty of an estimate from data analysis. The 95% credible intervals are calculated so that there is a 95% probability of the true value lying in the interval. A wider interval indicates more uncertainty in the estimate. Overlapping credible intervals indicate that there may not be a true difference between two estimates. For more information, see our methodology page on statistical uncertainty.

The Nuffield Department of Medicine study protocol specifies the research for the study.

The Welsh Government publishes results from the COVID-19 Infection Survey in English and results from the COVID-19 Infection Survey in Welsh that describe COVID-19 infections in Wales.

The Department of Health (Northern Ireland) publishes results from the COVID-19 Infection Survey that describe COVID-19 infections in Northern Ireland.

The Scottish Government publishes results from the COVID-19 Infection Survey that describe COVID-19 infections in Scotland.