Update to comparability between the Telephone-operated Crime Survey for England and Wales and the face-to-face Crime Survey for England and Wales

• The Telephone-operated Crime Survey for England and Wales (TCSEW) became operational on 20 May 2020; it was a replacement for the face-to-face Crime Survey for England and Wales (CSEW), which was suspended on 17 March 2020 because of the coronavirus (COVID-19) pandemic.

• The initial report published in July 2021 concluded that the main measures of crime (number of incidents and prevalence rates in the previous 12 months) were broadly comparable when certain adjustments were applied.

• The CSEW restarted in October 2021 before the TCSEW was closed in March 2022, providing six months of comparable data from both surveys for comparability analysis.

• Analysis showed that there were no statistically significant differences in estimates for most headline crime types, providing further evidence that the survey estimates are broadly comparable.

• The TCSEW may have been underestimating the level of vehicle-related theft offences while overestimating the level of computer misuse offences in comparison with the CSEW; these differences may be the result of attrition of respondents between survey waves and resulting differences in characteristics between the two samples.

• The increased number of computer misuse offences in the TCSEW may have also been the result of additional survey bias having specific effects on this topic, such as recall bias and panel conditioning, as a result of the shorter three-month reporting period and being asked the questions in multiple waves of the survey.

In our Comparability between the Telephone-operated Crime Survey for England and Wales and the face-to-face Crime Survey for England and Wales methodology, we were able to explore a range of factors that may have had a bearing on the comparability of estimates between the Telephone-operated Crime Survey for England and Wales (TCSEW) and the Crime Survey for England and Wales (CSEW). These included differences in survey and sample design, as well as mode and questionnaire changes. The report found that the main measures of crime (number of crime incidents in the previous 12 months) were broadly comparable when certain adjustments were applied.

The TCSEW was closed in March 2022, following the recommencement of the CSEW in October 2021, providing six months of data from both surveys that could be used for comparability analysis. This allowed us to investigate whether estimates derived from data collected from the two surveys over a concurrent time period were comparable or whether differences across the surveys had any unforeseen impacts.

Analysis showed that there were no significant differences in estimates between the Crime Survey for England and Wales (CSEW) and the Telephone-operated Crime Survey for England and Wales (TCSEW) for most headline crime types. This suggests that, overall, the initial conclusion that the main measures of crime (numbers of incidents and prevalence rates in the previous 12 months) were broadly comparable was correct (Table 1).

However, there were statistically significant differences in estimates of vehicle-related theft and computer misuse. The TCSEW may have been underestimating the level of vehicle-related theft offences while overestimating the level of computer misuse offences. There are likely to be multiple reasons for these differences stemming from the differences in survey design. These include attrition bias, recall bias, panel conditioning and mode effects.

Attrition bias

Attrition bias, whereby certain respondents are more likely to drop out of the study and differ in characteristics compared with those who remain in the study, is likely to have been an important factor. The TCSEW was only designed as a short-term solution, with possibly two to three waves of data collection, but this was extended to seven waves while face-to-face interviewing was not possible. This meant that the differences in characteristics between the two samples diverged further as more waves of data were collected. Although survey weights were calculated to account for these differences, it is possible that not all differences were captured and accounted for.

Recall bias and panel conditioning

The increased number of computer misuse offences in the TCSEW may have also been the result of recall bias and panel conditioning as a result of the shorter three-month reporting period and being asked the questions in multiple waves of the survey.

As both surveys ask about past events, they are both subject to recall bias as they rely on the respondents to accurately remember and recall past exposures and outcomes. Recall bias is related to a number of factors, including length of the recall period, characteristics of the outcome under investigation and study design (for example, cross-sectional or longitudinal).

For example, respondents are more likely to recall an incident accurately if it took place in the last three months rather than a year ago. This type of bias is likely to be more important for less serious offences, such as computer misuse, which may be more likely to be forgotten over a longer time-period in comparison with more serious offences, such as violence or domestic burglary.

In addition, if the same questions are asked at regular intervals, as in the TCSEW, this may stimulate respondents to remember less serious offences that they would not otherwise recall, making them more likely to be recalled in subsequent waves of the survey.

At the same time, it is possible that panel conditioning, whereby respondents can remember what happened in previous interviews, may have led to some respondents giving different answers to get the interview over quicker. This may explain why vehicle-related theft offences were lower in the TCSEW compared with the CSEW and, although differences across other crime types were not significant, why there were generally lower estimates across other crime types.

It is also possible that there is a stronger "telescoping" effect (a type of recall bias) for the CSEW, whereby incidents just outside of the reference period are placed within the reference period because it is a salient experience to the respondent. This may be more likely to affect the CSEW, where respondents only had one opportunity to describe their experience of crime because of its cross-sectional design.

Mode effects

Although the initial report explored the impact of modal changes on the comparability of estimates, these findings suggest that further research is required. While it is likely that attrition bias, recall bias and panel conditioning have affected estimates, it is possible that the change in mode has also had unforeseen impacts. This could include changes to the interaction between interviewer and respondent, respondent environment and social desirability bias.

Overall, the findings from this analysis should be interpreted with caution. Analysis was only based on six months of data that were collected over a comparable time period. In addition to the possible bias introduced by the TCSEW, it is important to note that the reintroduced CSEW estimates have not been classified as National Statistics. This is because of lower response rates, which impacts on the quality of the estimates. At the end of March 2022, we had an interim response rate of 39%. It is acknowledged that this is much lower than the historical CSEW response rate.

We plan on moving the crime survey to a multimodal panel design approach to make survey data collection more resilient to unforeseen future events. This will involve conducting face-to-face interviews with respondents and following them up after 12 months with a telephone interview. Although the panel design might introduce some conditioning and bias, this should be less pronounced over a 12-month period compared with a three-month period. In light of this analysis, further research is needed to ensure that we minimise attrition bias and try to understand if changes to our survey design will introduce additional bias because of repeated interviewing and mode effects, particularly for computer misuse offences.