Transformation of the Crime Survey for England and Wales - Discovery research on the redesign of multi-mode questions

The Centre for Crime and Justice (CCJ) in the Office for National Statistics (ONS) Crime, Income and Wealth (CIW) Division asked the Qualitative and Data Collection Methodology (QDCM) team in the ONS Methodology and Quality Directorate to undertake methodological research as part of the Crime Survey for England and Wales (CSEW) transformation programme. This work forms part of the ONS’s wider transformation strategy, which includes moving household and business surveys to mixed mode and/or online collection.

The aim of this transformation is to redesign the screener and victimisation modules of the questionnaire (used to measure incidence and prevalence of crime in the last 12 months). These will be employed in a mixed-mode design, where wave 1 is conducted face-to-face and subsequent waves by telephone (introduced in 2022). If shown to be feasible, an online collection mode, for wave 2 onwards will be designed, tested and introduced. The redesign will also make improvements that address issues with question design and complexity, including those that have been identified in previous research. Changes in question design and collection mode may result in disruption to the historic time series.

The design will aim to collect equivalent data from respondents in all modes and at all waves, minimising measurement error including those because of mode effects. It will also aim to provide a better survey experience for respondents and interviewers, while continuing to meet data user requirements.

Findings from this initial discovery phase research are based on:

• consideration of the design and content of the CSEW
• a review of previous research into CSEW online data collection
• primary research with CSEW interviewers and coders

The high-level conclusions we have drawn are:

• that there is much that can be done to build on previous research related to improving the questionnaire design, irrespective of mode
• that employing the online mode to collect all screener and victim form data would potentially work for most respondents, who have no or “simple” crime experiences
• however, online collection would be more difficult to design effectively for those with more “complex” crime profiles
• such complex crime profiles include multiple and/or repeat victimisation and where two or more crime types might be interlinked, where there is risk of double counting of offences that were part of a single incident, only one of which ought to be coded, according to the Principal Crime rule within the Home Office Counting Rules for Recorded Crime (PDF, 276KB)

Potential options for the future design include:

1) deciding to not employ online data collection, but proceed with improving the face-to-face and telephone questionnaires

2) continuing to investigate the feasibility of an online mode that includes respondents with complex crime profiles

3) focusing on development of an online CSEW design that works for most respondents, but in which the complex cases would be identified during the survey, to be flagged for referral to an interviewer to collect the remaining data

In addition to these options, we have set out several methodological topics and issues to research, many of which apply to all options, others being applicable to one or two.

One of the central aims of the transformation is to introduce an online mode to increase cost savings and survey accessibility. One of the main benefits of an online capability is to support self-completion modules for sensitive and non-sensitive topics, enabling the flexible rotation of existing topics and the introduction of questions to explore new areas of interest. Therefore, after considering the three options with the CCJ, the QDCM will aim to develop an online mode including complex cases, to try to realise these benefits, and because of potential practical complexities and data quality concerns around mixing modes within wave. These include attrition between online and interview mode, with potential loss of an important data element.

Therefore, research will continue based on option 2. The advantages of designing an online mode that can successfully measure crime incidence including “complex” cases make attempting to overcome the challenges worthwhile. The next stage will research the feasibility of measuring crime incidence online including respondents with complex crime profiles. Focus will be limited to the questions and methodological topics and issues that are most important for measuring crime incidence and offence coding.

This will lead to an assessment of the feasibility of option 2, which will set the direction of subsequent development work. If option 3 were then pursued, research would seek to establish the threshold or types of complex cases which cannot be collected online and whether mixed-mode collection within wave is feasible.

Planning will be informed by overall CSEW transformation timescales and plans, priorities and resource availability.

As part of the recent Office for National Statistics (ONS) transformation strategy, the Crime Survey for England and Wales (CSEW) has changed from a cross-sectional, face-to-face design to a longitudinal panel survey conducted at 12-month intervals, with a face-to-face interview at wave 1 and subsequent waves by telephone. The long-term aim is to supplement collections from wave 2 using an online survey, subject to an assessment of feasibility.

Our aim for the redesign is to collect accurate and equivalent data from respondents in all modes, to meet data user requirements and minimise mode effects on measurement error. This will improve the respondent and interviewer experience of the survey. The scope of our transformation work focuses specifically on the redesign and qualitative testing of the screener and victimisation modules; the core parts of the questionnaire which collect the main estimates of crime incidence and prevalence.

This discovery phase comprises exploratory research informed by a Respondent Centred Design Framework (RCDF), as recommended to achieve a "gold", "silver" or "bronze" standard survey transformation in ONS guidance due for publication in May 2023. CSEW transformation will aim to meet the gold standard.

The aims and associated activities were to:

• understand the CSEW and its data user needs by reviewing different iterations of the questionnaire, technical reports, coding process and counting rules documents and our 2022 Consultation on the redesign of the Crime Survey for England and Wales
• identify known issues with the survey and potential opportunities and risks by reviewing the research undertaken by Kantar Public in 2017 to 2018 and 2022 into the feasibility of an online mode
• conduct our own qualitative research with Kantar Public interviewers and coders to gain insight from their experiences regarding the administration and offence coding of the survey in different modes
• examine data from the CSEW, including frequencies, cross-tabulations and rates of “missingness” to gain insights into issues with questions and to inform changes to questions
• review research findings and the extent to which they support previous research, including Kantar Public’s literature review of 15 international crime surveys which have undertaken transformation work, to help build a comprehensive overview
• review the US National Crime Victimisation Survey (NCVS), which operates as a panel survey with predominantly face-to-face interviews at wave 1 and telephone follow up interviews at six monthly intervals
• use all the evidence gathered to assess the issues related to designing a mixed mode and longitudinal questionnaire, including assessing the feasibility of an online mode from wave 2 onwards
• identify important decisions that need to be made by the Centre for Crime and Justice (CCJ) team and set out options for consideration

This section summarises the high-level findings across Kantar Public’s development work and our discovery research with Kantar Public interviewers and coders.

• Kantar Public developed an online instrument which works for most respondents, that is, those with a “simple” experience of crime, or no experience, but problems arise for victims with “complex” crime profiles as well as the accurate capture of fraud crime data.
• Our focus groups with Kantar Public interviewers highlighted several issues with the Crime Survey for England and Wales (CSEW) questionnaire, and the important role of interviewers in resolving these issues, for example, double counting, difficulty answering fraud questions and determining whether crimes are part of a series.
• Research sessions with Kantar Public coders who were involved in coding the 2022 quantitative trial data showed that the telephone mode worked well, but complexities arose when coding fraud crimes from the online mode, mainly because of the lack of information provided by respondents. We identified issues with duplication of victim forms and clear satisficing (suboptimal responses being given because of shortcutting of response processes).

Kantar Public’s research into the feasibility of an online mode

Kantar Public were commissioned by the Office for National Statistics (ONS) to investigate the feasibility of collecting CSEW data via an online self-completion questionnaire, with a focus on the redesign of the core sections.

Their 2017 to 2018 work on the re-design of Crime Survey for England and Wales (CSEW) made significant progress towards producing an online self-completion instrument which could work in the field for a large proportion of respondents. These respondents included those who had experienced no crime or experienced a simple or singular incident. For example, in common victim scenarios, such as theft of a car or an assault, the online questionnaire was quick and easy to complete.

However, problems arose for respondents with complex crime profiles, including experience of:

• multiple or repeat victimisation
• an incident which involved more than one crime and therefore susceptible to double counting

The complex series of check questions and validation screens that were used to disentangle the circumstances of these crimes and to avoid double counting, were found to be too cognitively challenging for many respondents. Accurately capturing and counting fraud was also problematic because of the complexities in the crime classification rules, which are different for fraud and traditional crimes (those against the person or to property).

Counting and classifying incidents in the CSEW is complex. Interviewers who have worked on the survey have developed strategies for managing this complexity and resolving errors and inconsistencies in the field. Translating this into an online self-completion instrument presented several challenges. While prevalence is relatively straightforward to measure in any mode, the measurement of incidence, repeat and multiple victimisation is much more complex and does not easily translate into a respondent-focused self-completion survey. These questionnaire complexities therefore introduce clear risks in the context of an online survey, for example measurement error, higher break-off rates and reduced respondent engagement.

Kantar Public developed and conducted a live trial of an online version of the CSEW questionnaire as reported in Research on Transforming the Crime Survey for England and Wales, October 2022 (PDF, 1,884KB). This built on their previous work and included a quantitative trial to determine the impact of an online data collection mode on measurement, and how this differs from the telephone mode. This provided further insight into the nature and scale of the challenges.

There were no examples of crime surveys in the international review that Kantar Public conducted as part of this work which replicated the full complexity of the CSEW in any mode. The complexity of the CSEW includes measuring incidence as well as prevalence and differentiating series from multiple incidents.

In Kantar Public’s trial some issues were more prevalent in the online mode compared with telephone, including:

• accuracy of date recall
• potential underreporting of sensitive crimes
• increased out of scope offence codes, for example, personal crimes that happened to someone else or incidents of fraud where the respondent was not the “Specific Intended Victim” (SIV) that is, they did not respond to the fraud attempt

Issues for multiple and repeat victims, for example with double counting or duplication of crime incidents (the complex series of checks in the online mode to resolve issues in the absence of an interviewer were cognitively challenging for respondents). The definition of “series” crimes is complex for respondents who have experienced multiple incidents of the same crime type because the line between what is “similar” and “different” can become blurred.

Focus groups with Kantar Public interviewers

We held two focus groups with CSEW interviewers from Kantar Public with the aim of gaining insight into how the administration of the screener and victimisation modules work in practice. This identified potential issues with the current modes (face-to-face and telephone) in recording complex crimes and interviewers’ views on the proposed transformation of the survey.

Our findings aligned with many of the issues identified in Kantar Public’s research. Several themes arose, including issues with the survey structure, length, wording of questions and recall bias. The important role of interviewer moderation to enable accurate data collection and maintain respondent engagement was apparent. For example, resolving issues such as:

• double counting
• gathering sufficient information from open descriptions to reduce survey length and improve flow
• back tracking through the survey to amend screeners based on new information in the victim form
• clarifying question misunderstandings

Although the importance of interviewer moderation was highlighted, one theme we identified was respondent hesitancy in reporting sensitive crimes or being a victim of fraud to the interviewer, which may potentially lead to an undercount in interviewer-modes. However, it is worth noting that respondents in Kantar Public’s quantitative live trial were more likely to skip sensitive questions in the online self-completion mode when given the option compared with an interviewer-administered survey.

Exploration of CSEW offence coding processes with Kantar Public coders

As part of our discovery work, it was important to gain a detailed understanding of the offence coding process to ensure the redesign of the CSEW does not adversely affect this and, where possible, improves it. Our high-level findings from research sessions with members of the coding team are:

• overall, the change from face-to-face to telephone mode did not affect the coding process during the coronavirus (COVID-19) pandemic as the telephone survey was designed to obtain important information required for coding
• having coded both the telephone and online versions of Kantar Public’s redesigned questionnaire (for their live trial), coders felt that the questions worked well in both modes and were easy to code
• while coding of the online mode used in the live trial generally worked well, there were complexities with coding fraud offences
• insufficient information provided by respondents in the online open description question and inconsistencies between this and the closed questions in the victim form caused issues with coding fraud online including difficulties determining whether or not the respondent was the SIV
• the lack of information provided by respondents in relation to fraud may be because of greater difficultly describing these incidents compared with “traditional” crimes and/or lack of clarity about how much information is required given the absence of an interviewer to probe respondents
• duplication of fraud crimes was an issue with the online mode, with respondents sometimes filling out multiple forms for a series of incidents, leading to clear satisficing

The coders made various suggestions for the redesign of the victim form, to improve the quality of data and reduce the complexity of coding fraud offences.

Options for consideration by the Centre for Crime and Justice (CCJ)

The work undertaken during this discovery phase has enabled the Qualitative and Data Collection Methodology (QDCM) team to understand the Crime Survey for England and Wales (CSEW) and the challenges of developing an online mode.

Developing an online instrument to replicate the full complexity of the existing screener and victimisation modules will be challenging and might prove to be unfeasible. We have come to this conclusion based on Kantar Public’s research testing the feasibility of an online mode, and our findings from the discovery work with interviewers and coders. The quality of data could be compromised by various factors including the impact of modal differences, impacting the ability to measure incidence and code crimes accurately.

However, one of the central aims of the CSEW transformation is to introduce an online mode to increase cost savings and accessibility of the survey. One of the main benefits of an online capability is to support self-completion modules for both sensitive and non-sensitive topics, enabling the flexible rotation of existing topics and the introduction of new areas of interest.

Three options were proposed for future research for consideration by the CCJ. Throughout the options we refer to “complex” crime profiles. This refers to:

• multiple victimisation, a person or household that has been victimised on two or more separate occasions in a given period of time
• repeat victimisation, a sub-set of multiple victimisation where a person or household has been victimised on two or more separate occasions in a given period of time, and where the crime type is the same
• multiple feature crime, a person or household has experienced an incident where a positive response could have been given at two or more screener questions

However, we will define what is “complex” in further work.

Option 1: Do not continue to develop or test for an online mode

Research would focus on improving the CSEW face-to-face and telephone questionnaires, conducting an “expert review” of the questions and incorporating findings from across the previous work. Kantar Public’s redesigned specification for both an online and interviewer mode, taking account of known issues with the current face-to-face survey, could be used as a starting point for the face-to-face and telephone modes. Work would include resolving the issues identified by previous research, as well as the measurement of violent crime. Improving the design might still require significant changes to Kantar Public’s revised questions.

Not having an online mode comes with risks, for example any potential cost savings from the online mode, gains in response rates and reduction in response bias from wave 2 onwards, would not be realised. It would also have an impact on the development of a panel design that can accommodate flexible rotation of modules and introduction of additional survey content.

Option 2: Aim to develop and test the feasibility of an online mode which work for all respondents (including victims with “complex” crime profiles)

The design of an online mode would require simplification of the highly complex design of the CSEW. This option could involve taking an alternative approach to the questioning, informed by mental models research. Mental models are how people conceptualise topics and terms and process them. Common methods include card sorting and “think aloud” techniques.

This option would also take into consideration Kantar Public’s redesigned instrument including the more complex recommendations arising from the trial. Some of Kantar Public’s recommendations apply to an online mode only, for example, the use of automated crime description tags and a video to explain concepts. We would need to consider how such recommendations might work in a mixed-mode design when optimising for telephone and face-to-face modes.

This option would require a significant amount of time for development and testing. Kantar Public suggest that sufficient time needs to be allowed to fully develop and iteratively test the content and design of a new multi-mode survey design and acknowledge that this work would be “exceptionally challenging.”

Option 3: Focus development work on an online mode that works for most respondents, but those with “complex crime” profiles answer part of the questionnaire for that wave in interviewer-mode

A vast majority of the complications Kantar Public found with introducing an online mode (for example, survey length, cognitive burden because of question complexity, double counting and satisficing) are for victims with “complex” crime profiles. One of Kantar Public’s suggestions was to consider “flagging” respondents with complex crime histories so that they complete the questions for that wave in interviewer-mode.

Option 3 would potentially be more straightforward in resolving some of the question design issues than option 2. This option would enable us to make use of the benefits of an online mode, while avoiding its pitfalls by ensuring the complex cases are properly moderated and issues resolved in the presence of an interviewer. This option might require less development time than option 2, as Kantar Public have developed an online instrument which works well for most respondents with a “simple” crime profile.

Work would ensure the mixed mode questionnaire works for all respondents, with the online element focused on those with a “simple” experience of crime. Changes would be implemented to enable accurate offence coding in the absence of an interviewer, for example, improvement of the fraud victimisation module and open description. An important research objective would include establishing the criteria according to which participants could be flagged and redirected, for example, the data required to determine what constitutes “complex crime” (especially from a respondent perspective). Consideration would be given to how easily an interviewer would be able to resume from where the online mode was exited, and whether they would recap or check what the respondent had answered online.

We acknowledge potential complications relating to switching modes within wave for an individual. Risks to respondent engagement and attrition could impact data quality with regard to this important analytical subgroup. A strategy to manage the transition and minimise drop out between modes within wave would need to be considered carefully.

Methodological issues and topics to address

Further to our broad design options, from the discovery work we have identified various methodological issues and topics to be addressed and potentially researched further.

Confirmation of data requirements and detailed review of Kantar Public’s redesigned instrument

An assessment of how well questions meet data user requirements, at face value, and confirmation that all questions are still required for outputs. That is, the effectiveness of questions to produce data that enables accurate measurement of crime prevalence and incidence and coding of offences according to Home Office Crime Counting Rules, and obtaining data required for other outputs. It is important to gain clarity on which questions in the CSEW are still required for data collection.

Structure and ordering of questions

Respondent mental models might not always fit with the current screener approach. Consideration should be given to redevelopment of the questionnaire structure in response to mental model research. For example, the finding that people think about crimes as holistic incidents rather than chronologically or in terms of the different features of incidents.

Interviewer suggestions to collect more information upfront to avoid backtracking through the survey to correct the screeners could be tested. Mental models research could explore the most effective ordering of screeners to compare with the existing scale from less to more serious crime types. Research would explore how the use of “ask or record” in interview modes can be precluded in the design of questions for online mode, where this would not be an appropriate method.

Double counting

The check questions incorporated into Kantar Public’s online instrument to identify and correct double counting (within wave) were considered burdensome and confusing for respondents. To mitigate against these risks, an alternative method will be developed and tested. Consideration will be given to Kantar Public’s recommendations and suggestions, including:

• allowing respondents to “bypass” the victim form in the case of duplication
• using more “composite” screeners to ask about incidents that tend to be linked (as they are similar) in the same screeners has been trialled for the violence screener and could be considered for thefts both inside and outside the home (for example, all incidents relating to a respondent’s property); fraud related crimes could also be grouped as testing found that the uncertainty surrounding how incidents of fraud occur, make linking them to a specific screener more difficult
• restricting double counting checks to only cross-reference an incident against previously mentioned incidents that took place in the same month, rather than against all reported incidents

De-duplication of victim modules

Our research with both interviewers and coders highlighted issues with the inability to skip a victim form that relates to an incident that has already been covered. Kantar Public suggest a check question at the start of the victim module which would allow respondents to bypass a victim module if they had already answered questions about the same incident in a previous victim module.

Multiple and repeat victimisation

There is cognitive burden for multiple and repeat victims because of increased length of the survey and additional complexities. Work to simplify the questionnaire and reduce the cognitive burden associated with double counting checks will be required. Consideration will be given to Kantar Public’s specific recommendations for reducing burden for these respondents online (relevant to option 2).

Series crimes

When a respondent has experienced more than one incident associated with the same screener, they are asked if these incidents were “similar” in nature, which some find complex. Consideration will be given to Kantar Public’s suggestion to avoid asking about “similar” and “separate” crimes and to instead apply a cut-off, for example, to always treat one to two crimes as separate and to always treat three or more incidents as a series.

Interviewers in the focus groups brought up other issues with the fact that only the most recent in a series of incidents is captured in the victim form, as the most recent incident may not be representative of the respondent’s overall experience. This suggests the need to explain to respondents why data on the last incident alone are collected. While a respondent’s latest incident may not be representative of others in the series, the purpose of the question is to capture an average that is representative across victims.

Flagging respondents with complex crime profiles

The point within the questionnaire (from wave 2 onwards) at which participants could be flagged and redirected from online to interviewer mode, would require scoping out and testing. Consideration would be given to whether an interviewer would be able to pick up from where the online mode left off and whether they would recap or check what the respondent had answered online (relevant to option 3 only).

Incident description open question

The open description question works well in interviewer mode and interviewers and coders in our research reported that it is “essential”. Consideration will be given to the interviewers’ suggestion to include it earlier in the victim form. If designing for online mode, the victim forms and open description (especially for fraud) require redesigning to overcome lack of information provided in the absence of an interviewer. Kantar Public coders suggested various solutions during the research sessions, which require further thought and testing.

Short-hand crime tags in the online form

These refer to codes used as text substitutions in Kantar Public’s online instrument, to indicate which incident type is being referred to in the victim module. These did not always provide accurate representations of the incident so would require further work to develop for an online form, if they are still considered useful within a more radical redesign.

Wording of the questions

One of the main findings from the focus groups with interviewers was the occasionally complex language of the questionnaire which they themselves struggle to understand and explain to respondents. The fraud victim module was identified as being particularly challenging, as well as other specific questions.

Question design without relying on help

Respondent and/or interviewer uncertainty as to question meaning poses a risk to data quality. The question redesign work we have outlined aims to avoid reliance on instructions, guidance and interviewer assistance, as per the Respondent Centred Design Framework (RCDF) component “design without relying on help”.

Kantar Public recommend developing an animation or video for the online form, to explain concepts to respondents at the start of the screener section. These would be supplemented by avatars or similar, which appear when a further survey definition, concept or reminder needs to be conveyed. The need for such a feature would be part of the consideration of designing without relying on help. The crucial need remains for clarity about question meaning, individually and collectively.

Longitudinal element and recall bias

Potential issues exist around the 12-month reference period and respondent recall of when crimes occur, such as telescoping effects, which could lead to double counting across waves. Research will consider how that can be mitigated. We will also consider the use of the Life Events calendar (a physical handout given to respondents to assist with the recall period) and adaptation of it to an online mode.

Specific question topics

Fraud and computer misuse

Fraud and computer misuse has been found to be complex for respondents to understand. Additionally, the nature of fraud and computer misuse offences evolve over time with the introduction of new technologies and methods. There is a need to design a set of questions that are independent of new technologies and methods.

Potential underreporting of sensitive crimes

Consideration will be given to how best to identify, design and order sensitive questions so that respondents can answer without fear of consequences or embarrassment, in any mode.

Violent crime

Harassment will be considered as part of the transformation of the CSEW but is outside the scope of QDCM’s discovery research. Consideration will be given to how best to capture all assaults including attempted and common assault. Qualitative research, for example mental models to understand how respondents define and categorise their experiences of violent crime, will be developed. Consideration will be given to Kantar Public’s recommendation of a separate module to include threats and harassment, to improve the relevance of the question sequence for victims of these incidents.

Sexual assault

We will review the effectiveness of the sexual assault screener and consider the extent to which it may be subject to measurement error, such as social desirability bias, across modes. If retained, we will explore which mode(s) are optimal for the capture of these data.

Attempted crime

The method by which attempted crimes are recorded at the screener questions needs further consideration. A possible approach would be to ask about actual and attempted crime in a single screener question rather than try to isolate these at the outset. Determining whether an incident was an actual or attempted crime could then be resolved later in the victim module.

Attrition within wave

There is a risk of attrition between the online survey and the interviewer-led follow up (relevant to option 3). A proportion of those who drop out might be victims of abuse, for example, domestic abuse cases. This may lead to increased bias in a subpopulation already difficult to measure and perhaps already underrepresented. Consideration could be given to alternative methods of keeping in contact with respondents and/or collecting data in a safe and private way, such as a chat function.

Review of the options

After consideration of the three options with the CCJ, we have agreed to develop an online mode to potentially realise the benefits outlined previously.

Research will continue based on option 2 as the advantages of designing an online mode that can successfully measure crime incidence including “complex” cases make attempting to overcome the challenges of this approach worthwhile.

Should findings indicate an online mode for all respondents unfeasible, option 3 would be pursued. Research would focus on resolving the practical and data quality concerns surrounding mixing modes in-wave, such as attrition between online and interview modes. We would also seek to establish the threshold or types of complex cases which cannot be collected online and whether mixed-mode for an individual respondent within a wave is feasible.

Transformation provides an opportunity to fundamentally improve the questionnaire, which has not significantly changed since its original design approximately 40 years ago.

An overall redevelopment for all modes is more desirable than to merely adapt the face-to-face Crime Survey for England and Wales (CSEW) for use in online mode and will enable us to go beyond the scope of Kantar Public’s work. This may result in significant changes to the questions, structure and feel of the survey, but would achieve a design that better reflects respondent needs and society today.

Changes in question design and collection mode may result in disruption to the time series. On completion of the survey redevelopment, a parallel run will compare data from different modes. The main objective, however, is to meet the data requirements (as confirmed) and improve measurement quality.

CSEW transformation will aim to meet the gold standard as outlined by the Office for National Statistics (ONS) Respondent Centred Design Framework (RCDF). It is anticipated that this process could take a significant amount of time, involving a multi-phase research programme to fully explore the many challenges faced by this redesign.

The next stage of CSEW questionnaire development will focus on research to assess the feasibility of measuring crime incidence in online mode including respondents with complex crime profiles, as per option 2. Given the number of methodological issues and topics that have been identified, we need to prioritise and provide a manageable focus. The scope will be limited to those questions required for measuring crime incidence and offence coding. This will include the content of the current screener module and parts of the victim form, omitting, for now, the questions in the victim form needed only for secondary outputs, such as the nature and cost of crime tables. However, there are some detailed issues or topics that are relevant to the measurement of incidence that it may not be possible to explore in depth in the first phase of research.

Planning the work will need careful consideration of research priorities. Organisation of the elements into work packages, sequencing and dependencies and use of an agile and iterative approach, will be considered. Initial ideas on how to conduct this next stage include two phases, as follow.

Discovery part 2

• This would include activities related to the first two items in the list of issues and topics we have discussed: reviewing and confirming data requirements and exploring how best to structure the questioning required for capturing incidence and detail required for offence coding. The latter would include reviewing Kantar Public’s redesigned questionnaire and their further recommendations and suggestions, as they relate to the structure and ordering of the survey.
• Considering the findings from the National Crime Victimisation Survey (NCVS) testing, including their “non-interleaved” approach, where follow up questions were asked at the start of the Crime Incident Report to help correctly classify crimes.

It would also involve designing with data: examination of CSEW data will provide insights into issues with questions and response options, such as high rates of missingness or low rates of selection. Past CSEW data will also inform the creation of user journeys, various personas of “complex crime” victims and sensitive crime victims. For each of these, we would track them through the questions to identify how we would want to capture their experience and identify points at which data collection problems may occur. This analysis would also provide metrics on the relative size of subgroups with more or less complex crime profiles, which might inform the feasibility assessment.

The ONS’s transformation guidance is due for publication in May 2023, as “Levels of Respondent Centred Design; options for materials and questionnaires Version II” (Hales and Wilson, 2023). It states that a minimum requirement for development to be respondent-centred and considered “transformation”, is that it must include research into respondent mental models, including in the discovery phase. Otherwise, it will be considered “continuous improvement”. We would carry out mental models work to build on Kantar Public’s research where participants were asked to summarise their experience over the last 12 months. We would focus on the following issues for online feasibility.

Order, structure and wording of the questionnaire

Respondent mental models do not always fit with the current screener approach and interviewers play an important role in resolving issues relating to this. This research will aim to assess the order of screeners which currently ask about crimes in order of least serious to most serious. We would encourage participants to use a think-aloud method to tell us about incidents they have experienced, and to better understand how they would naturally think and articulate it. This will help produce a more logical survey structure that feels more aligned with the respondents thought processes.

Use of the open description in the victim form

Mental models can explore how we could change the online victim form to be able to collect the more detailed, accurate information required for coding.

We will hold redesign workshop(s) or otherwise consult with colleagues in the CCJ, Kantar Public and other stakeholders as appropriate. This would add expertise from our colleagues and stakeholders to the insight and understanding gained from the research activities.

Specification, programming and “cogability” testing of redesigned questions

Based on the discovery part 2 findings and the challenges related to question wording listed among the issues and topics discussed, a new set of questions will be designed. Questions will be programmed in a software tool appropriate for qualitative testing and several rounds of iterations of cognitive and usability testing will be undertaken.

We will assess the priority of specific issues and topics and will identify which are discrete and which are inter-related and not easily separable. Depending on findings, remaining issues and topics might then be explored incrementally in further stages, leading to a gradual development of questions within the victimisation form.

At the end of this stage, we will conclude the feasibility assessment of option 2. Findings may be indicative rather than definitive and if so, further iterations will be needed. The criteria for assessment will be developed; for example, what defines a profile as being “too complex” to capture online, and estimation of the size of the complex profile respondent pool based on past survey data.

Work will be aligned to the RCDF principle of designing for online first, despite it being intended that the online mode would only be used from wave 2 onwards.

Potential subsequent development

This section outlines subsequent CSEW development that could follow the assessment after stage one. If feasibility of option 2 is demonstrated, we could proceed to design questions for secondary outputs and research into the issues and topics that were not relevant to incidence or complex crime; and optimise the online questions for interview modes.

If feasibility is not demonstrated, we will consider if research should pursue option 3, mixing mode for an individual respondent within wave, from wave 2 onwards. For example, identifying which crime profiles could be asked online and which could not, whether logistical constraints or complexities and/or concerns around within-wave attrition and data quality preclude this.

If options 2 and 3 are not considered feasible, the CCJ and stakeholders must assess the relative importance of measuring incidence against other benefits of introducing the online mode, including reduced survey costs and flexibility to add modules. For example, deciding whether, when and how to collect incidence (not at all, or not from wave 2, or only by telephone from wave 2 onwards). If measuring incidence at all waves is confirmed as the highest priority, consideration could then be given to pursuing option 1. Research will then focus on improving face-to-face and telephone modes and designing a questionnaire to collect screener and victim form data in a mixed mode and longitudinal design which minimises mode effects.

The various research pathways proposed will require periodical stages of pause-and-review to enable decisions about subsequent direction of the research. The inclusion of quantitative testing will be considered, for example, to provide measures of incidence or other outputs for comparative purposes and to analyse modal and longitudinal effects. Other aspects of the survey process that fall outside the scope of this work will also need consideration, such as other question topics or modules, respondent communications and field strategies, respondent full end-to-end journeys, response rates and biases, and attrition between waves.