Qualifications question development for Census 2021

In December 2018, the government presented to Parliament a White Paper Help Shape our Future: The 2021 Census of Population and Housing in England and Wales. This outlined the Office for National Statistics's (ONS's) proposal to collect information on qualifications to provide the basis for addressing variations in levels of skills and to enable interventions to be targeted appropriately.

In Census 2021, respondents will be asked questions about:

• apprenticeships

• degree-level or higher qualifications

• NVQ or equivalent qualifications

• A and AS level or equivalent qualifications

• GCSE or equivalent qualifications

• other qualifications (asked only if respondents have not selected any previous qualifications)

As in previous censuses, the question will only be asked of people who are aged 16 years or older.

The main changes to question design from 2011 are:

• the question has been changed from one long question into several shorter questions

• there are standalone questions on apprenticeships and degree-level and above qualifications

• we have simplified the list of qualifications shown to respondents

• we have added instructions and guidance text to help respondents answer the questions

Since the publication of the White Paper, the Office for National Statistics (ONS) has conducted and concluded the final phase of testing on qualifications. This report provides links to previously published research and the findings of additional testing that led to the final recommended questions for Census 2021 in England and Wales.

The questions and response options for Census 2021 have now been finalised through the census secondary legislation: The Census (England and Wales) Order 2020 and Census Regulations for England and Wales.

The evidence base for the recommendations made in the White Paper is discussed in Section 4: Research that led to the 2018 White Paper recommendations. The evidence base for the finalisation of the questions for Census 2021 is discussed the Section 5: Research that led to the recommended Census 2021 question designs.

In June 2015, the Office for National Statistics (ONS) held a formal, 12-week consultation process asking census users for their views on the topics that were required in the questionnaire in England and Wales. The aim of the consultation was to promote discussion and encourage the development of strong cases for topics to be included in Census 2021.

In May 2016, the ONS published its response to the 2021 Census topic consultation. This set out our updated view on the topics to be included in Census 2021, including:

• a summary of proposals for new topics

• next steps

• an overview of our plans

The 2021 Census topic consultation revealed a clear requirement for information on qualifications to understand the educational needs of the population, allocate resources and help inform policy.

A detailed summary of the consultation on education and qualifications can be found in the education topic report (PDF, 619KB). In this report, the ONS made clear commitments to the public. This included a commitment to shorten and simplify the qualifications question and to explore the use of administrative data on qualifications as a replacement for collecting such information in censuses and surveys.

We have provided an update on how we met these commitments in Annex 1 of this report.

Following this, we began a comprehensive programme of research and development. We have provided a full list of the tests used in the development of the qualifications questions in Annex 2. Further details are provided in the Summary of testing for Census 2021.

The tests utilised a range of qualitative and quantitative research methods. A short description of the different research methods and sampling techniques is given in the Question and questionnaire development overview for Census 2021. Testing included respondents from a wide range of educational backgrounds, for example, people with no qualifications, foreign qualifications, vocational qualifications, apprenticeships and professional qualifications.

In February 2018, we conducted a survey of stakeholders to further understand their data needs and to refine the requirements of the qualifications question.

In December 2018, the government presented to Parliament a White Paper Help Shape our Future: The 2021 Census of Population and Housing in England and Wales. This outlined our proposal to simplify the question on qualifications. Our research before the White Paper had shown that the user needs could not be met with a shorter question.

The question recommendations for Census 2021 are now finalised. We have evaluated the question for its potential impact on data quality, public acceptability, respondent burden, financial concerns and questionnaire mode. We have presented details of this evaluation in Annex 3.

References to tests take the form (Year: Test number). "Year" refers to the calendar year the test was undertaken. For example, the fifth test conducted in 2017 would be referenced as (2017:5).

Throughout development, the questions have undergone significant user experience (UX) testing (2017:2, 2018:2, 2019:1, 2020:2). UX testing focuses on understanding user behaviours as people interact with online services. Through observation techniques, task analysis and other feedback methodologies, it aims to develop a deep knowledge of these interactions and what it means for the design of a service.

Review of the qualifications question from the 2011 Census

We began our research for Census 2021 by looking at the qualifications question from the 2011 Census. The question had 13 response options, and in England it was:

Which of these qualifications do you have?

Tick every box that applies if you have any of the qualifications listed

If your UK qualification is not listed, tick the box that contains its nearest equivalent

If you have qualifications gained outside the UK, tick the 'Foreign qualifications' box and the nearest UK equivalents (if known)

[ ] 1 – 4 O levels/CSEs/GCSEs (any grades), Entry Level, Foundation Diploma

[ ] NVQ Level 1, Foundation GNVQ, Basic Skills

[ ] 5+ O levels (passes)/CSEs (grade1)/GCSEs (grades A* – C), School Certificate, 1 A level/2 – 3 AS levels/VCEs, Higher Diploma

[ ] NVQ Level 2, Intermediate GNVQ, City and Guilds Craft, BTEC First/General Diploma, RSA Diploma

[ ] Apprenticeship

[ ] 2+ A levels/VCEs, 4+ AS levels, Higher School Certificate, Progression/Advanced Diploma

[ ] NVQ level 3, Advanced GNVQ, City and Guilds Advanced Craft, ONC, OND, BTEC National, RSA Advanced Diploma

[ ] Degree (for example BA, BSc), Higher degree (for example MA, PhD, PGCE)

[ ] NVQ level 4 – 5, HNC, HND, RSA Higher Diploma, BTEC Higher Level

[ ] Professional qualifications (for example teaching, nursing, accountancy)

[ ] Other vocational/work-related qualifications

[ ] Foreign qualifications

[ ] No qualifications

The 2011 Census Quality Survey (PDF, 1.4MB) reported that the qualifications question had one of the lowest agreement rates (67.6%) of all the questions on the census. The agreement rate is calculated by comparing how a respondent answers the question on the questionnaire to how they answer during a face-to-face interview. A low agreement rate indicates that the quality of data collected is not ideal.

One reason for the low agreement rate was that many respondents found it hard to identify their qualifications, or equivalents, in the long and complex list presented to them. Another reason was that respondents found it difficult to recall their qualifications, particularly if they were achieved a long time ago.

Our report on the processing of the data from the 2011 Census (PDF, 819KB) found that the qualifications question had a relatively high non-response rate of 5.7%, suggesting that respondents may have found the question difficult to answer.

Our evaluation of the 2011 question in the education topic report (PDF, 619KB) found that the question had high potential for impact on data quality and respondent burden.

It was clear that there was a need to make the qualifications question easier to answer as part of the development of the question for Census 2021.

Refining the requirements from stakeholders

Although we had already consulted with stakeholders as part of the topic consultation in 2015, we also conducted a stakeholder survey in February 2018 to further refine their requirements.

The results of this survey showed that:

• stakeholders needed to identify groups with low or no qualifications

• stakeholders needed accurate data on apprenticeships but did not need to know the details on the type of apprenticeships

• very few stakeholders used data from the foreign qualifications response option

• shorter questions did not meet users' need for detailed data on qualifications

Changes to the paper questionnaire

When developing questions for Census 2021, we needed to ensure that the experience of answering questions on paper and online were closely matched to ensure that responses did not differ because of questionnaire mode.

In the early development phase of the qualifications question on Census 2021, the intention was to keep the paper questionnaire to four pages of questions per person, as it had been in 2011. This gave us limited scope to redevelop the question into a new format as we needed to ensure that any new question(s) fitted within the space available on the paper questionnaire.

Our early testing of questions (2016:6, 2018:1 and 2018:6) found that it was not possible to shorten the question on qualifications and to fully meet the user need for data.

In April 2018, a proposal to increase the length of the questionnaire to five pages per person was approved. The main reason behind this change was the introduction of new question topics; however, the early testing of shorter questions on qualifications also fed into this decision.

Recommendations for the qualifications question

The recommendations in the White Paper followed our research into the 2011 Census question and our consultation with stakeholders. These recommendations were to simplify the questions on education and to provide better quality data than that provided by the 2011 Census question as well as ensuring the data provided meet the needs of data users.

Following the increase in space available on the paper questionnaire and the recommendation in the White Paper, we began redesigning the qualifications question.

Splitting the question into sections

It was clear that one of the issues of the qualifications question from the 2011 Census was that it was one question with many response options that contained a lot of text and abbreviations. To simplify the layout of the question, we decided to separate the question into sections, grouping similar levels of qualifications together. This change had also been recommended following feedback from previous testing (2016:6 and 2017:18).

We started by developing and testing different versions of the question to be used online and on paper (2018:14).

We knew from stakeholder engagement that data about apprenticeships was needed, so we created two versions of a standalone question on apprenticeships, both with a "Yes" or "No" answer:

• Have you achieved an apprenticeship?

• Have you completed an apprenticeship (trade, advanced, foundation or modern)?

We grouped all the other qualifications using these headings and examples:

Higher Qualifications

• degrees or above, NVQ level 4 and above, higher diplomas (HND), higher national certificate (HNC), professional qualifications, RSA higher diploma, BTEC higher diploma, other higher qualifications or foreign equivalents

Advanced Qualifications

• 2 or more A levels, 4 or more AS levels, NVQ level 3, advanced GNVQ, RSA advanced diploma, City and Guilds advanced craft, ordinary national certificate (ONC) or diploma (OND), BTEC national, Advanced Welsh Baccalaureate, other advanced qualifications or foreign certificate

Intermediate Qualifications

• 1 A level, 2 or 3 AS levels, 5 or more GCSEs (grade A*-C), NVQ level 2, intermediate GNVQ, City and Guilds craft, RSA diploma, National Welsh Baccalaureate, other intermediate qualifications or foreign equivalents

Foundation Qualifications

• fewer than 5 GCSEs (grade A*–C), NVQ level 1, Foundation Welsh Baccalaureate, other foundation qualifications or foreign equivalents

When we began developing the questions for online and paper, it became clear that we needed to present the questions differently in each medium. On paper, we were constrained by the space available, whereas online we were able to follow best practice for online design and follow the Government Digital Service's Service Standard. However, we also needed to ensure that data are captured in a consistent way in both mediums so that outputs are comparable.

Developing the instruction

On paper, we added an instruction prior to the qualifications questions. Online, the same instruction was provided as an interstitial page.

Evidence from the user experience (UX) testing led to development of this instruction. Issues included respondents often only recording their highest qualifications, as they assumed this was all that was wanted, and the instruction to include qualifications achieved in England, Wales or elsewhere was considered too vague.

Therefore, we developed the instruction "The next set of questions is about any qualifications you have ever achieved in England, Wales or elsewhere, including equivalents, even if you are not using them now." Results from testing the interstitial page (2018:39) showed that the wording was well understood but that "elsewhere" should be changed to "worldwide" to help understanding, and the instructions helped those with foreign qualifications understand they should include them.

Electronic questionnaire

Online, each level of qualification, for example Higher Qualifications, was a standalone question on a separate screen.

Version 1: The questions were displayed in decreasing order of level of qualifications, from Higher to Foundation Qualifications. Participants were asked if they had, for example, any Higher Qualifications, and the qualifications were listed as response options and the participants were asked to select all response options that applied to them. There was also an option for participants to record that they did not have any qualifications at that level.

Version 2: This version had the same order of questions and routing as Version 1; however, the list of qualifications at each stage was simplified so that there were fewer response options. There was also a guidance instruction, "Further questions will be asked if you don't have any qualifications at this level", after each question.

Version 3: This version was similar to Version 2 but with the order of questions going from lowest level of qualifications to highest level.

Paper questionnaire

Version 1: The questions were displayed in decreasing order of level of qualification, from Higher to Foundation. Participants were asked if they had any qualifications at that level and given examples of qualifications. They answered "Yes" or "No". For example, the question on Higher Qualifications was:

Have you achieved any Higher Qualifications?

Include degrees or above, NVQ level 4+, higher diplomas or certificates, other level 4+ qualifications

[ ] Yes –> Go to [8]

[ ] No

If participants answered "Yes" to any question, they were routed around the remaining questions to the question about apprenticeships.

Version 2: Similar to Version 1, except the qualifications were presented in ascending order of level of qualification.

Results from all rounds of testing

Both versions of the standalone question on apprenticeships were well understood and easily answered by participants.

The results showed that all the versions of the other qualifications questions were relatively easy to answer for people with standard and recent qualifications such as GCSEs and A levels.

However, people with older or vocational qualifications found both hierarchical designs problematic, and they found it difficult to match their qualifications to equivalents. This problem was worse online where they could not see ahead to the full list of qualifications at each level.

People taking part online also tended to try to provide all their qualifications on the first screen presented to them by trying to match their qualifications to the nearest equivalent on screen. This was because it was not clear to them that further screens would show more options.

This led to respondents either overstating their qualifications (when shown qualifications from high to low) or understating their qualifications (when shown qualifications from low to high).

Changes to the design

Adding a degree-level or above question

Stakeholders had told us that they had a need to collect data on qualifications at level 4 and above. These are qualification such as HNCs (level 4), foundation degrees (level 5), NVQ level 4 and degrees (level 6+). However, there was not a data need to distinguish between levels above level 4. We decided to add a simple standalone question to capture these data.

Results from testing showed that respondents did not understand terminology used to describe levels of qualifications, such as "higher" or "level 4".

Since the most commonly held qualification around this level is a degree, the term is widely understood, and because testing had shown that people were accurate in comparing other qualifications to degree-level, we decided to use the term "degree" in the question. So, the question stem became, "Have you achieved a qualification at degree level or above?".

Grouping the qualifications

In our previous tests, we had described distinct levels of qualifications as the user need was for data about specific levels of qualifications. However, respondents did not fully understand this terminology.

In addition, evidence from the UX testing (2017:2 and 2018:2) showed that including lots of text on screen increased respondent burden. When respondents struggled with large amounts of text on a screen, they often missed response options that they should have selected.

To make the groupings easy to read, understand and answer, we changed them so that similar qualifications were grouped together, still allowing levels of qualifications to be collected.

At the same time, we conducted research to identify the most commonly held qualifications in the UK. We used this list to reduce the number of examples of qualifications within the question.

This resulted in the following groupings:

GCSEs or equivalent

• 5 or more GCSEs (A*–C, 9-4), O levels (passes) or CSEs (grade 1)

• Any other GCSEs, O levels or CSEs (any grades) Basic Skills course

AS, A level or equivalent

• 2 or more A levels, 4 or more AS levels

• 1 A level, 2-3 AS levels

• 1 AS level

NVQ or equivalent

• NVQ level 3, BTEC National, OND or ONC, City and Guilds Advanced Craft

• NVQ level 2, BTEC General, City and Guilds Craft

• NVQ level 1

In the English and Welsh language versions of the census for Wales, the list of qualifications is the same as for the census in England, except that there are three qualifications specific to Wales: "Advanced Welsh Baccalaureate" is available as an option in the A level section, and "Intermediate or National Welsh Baccalaureate" and "Foundation Welsh Baccalaureate" are available in the GCSE section. On the paper questionnaire, there is not enough space for the "Intermediate or National Welsh Baccalaureate" response option, so this has been changed to "Intermediate Welsh Baccalaureate".

On the electronic questionnaire, some of the qualifications have been moved into instructional text after the response option to simplify the main text of the response option. For example, the first response option for the GCSE or equivalent question is:

[ ] 5 or more GCSEs grades A* to C or 9 to 4

Include 5 or more O level passes or CSEs grade 1

Layout

The recommendation from testing was to keep the questions asking about NVQs and A-level and GCSE-level qualifications as separate questions. This layout, which we applied to the electronic questionnaire, conforms to the Government Digital Service's Service Standard and accessibility guidelines.

There was insufficient space to allow this on paper, so instead we used subheadings to group the qualifications together, as specified in the previous section.

Ordering the question

The standalone questions on apprenticeships and degree-level or above qualifications come first on both questionnaires. Since there was a user need to collect data on level 4 and above qualifications, which is the purpose of the degree-level question, this was placed directly after the apprenticeships question. Testing had shown that people tended to prefer to record their highest qualification first, and degrees are the most commonly held qualification at level 4 or above, so positioning the question after apprenticeships was considered optimal for data quality.

Online, the order of the rest of the qualifications is NVQ, A level, GCSE and any other qualifications. We put NVQs before the other qualifications as testing showed that if this question was later, then people tended to inaccurately record their vocational qualifications, trying to match them to other academic qualifications.

On paper, the order is GCSE, A level, NVQ and any other qualifications. On paper, where respondents can see all the questions easily, there was less of a problem of respondents recording their qualifications too early, or in the wrong place, and therefore using an ascending order of qualification was appropriate.

Response options

Testing had shown that sometimes respondents did not capture their qualifications accurately online as they could not see what the next question would be. To help respondents, we added an extra response option online, "None of these apply", to the NVQ, A level and GCSE questions, which includes text indicating what is asked in the next question. For example, the response option on the A level screen is:

[ ] None of these apply

Questions on GCSEs and equivalents will follow

We also added text to the "No" response option of the degree-level or above qualifications questions on the electronic questionnaire, to indicate that more questions would follow:

[ ] No

Questions on other NVQs, A levels and GCSEs or equivalents will follow

Another recommendation from testing was to add an option that allowed respondents to indicate they have other qualifications so that those who cannot match their qualifications to any of those listed are still able to answer the question. Therefore, we added two response options for other qualifications achieved in England or Wales and outside England and Wales. On paper, because of space constraints, there is only one response option.

Testing the new layout

We undertook further qualitative testing (2018:39) with members of the public who represented a broad range of educational backgrounds. This was followed by a quantitative test (2018:43) using the same question designs.

These tests used the layout that contained subheadings on paper, the simplified list of the most common qualifications, and standalone questions on apprenticeships and degree-level or above qualifications.

Participants saw the instruction "Include any formal qualifications you have ever achieved in England, Wales or elsewhere, including equivalents, even if you are not using them now" before the questions.

The questions on apprenticeships and degree-level or above qualifications were answered with "Yes" or "No".

Under the degree-level question, there was text to clarify what was meant by degree level or above: "For example, degree, foundation degree, HND or HNC, NVQ level 4 and above, teaching or nursing."

On paper, the question "Have you achieved any other qualifications?" was laid out under subheadings, with tick boxes for each option below the subheading. The subheadings and response options were:

GCSEs or equivalent

• 5 or more GCSEs (A*–C, 9-4), O level (passes) or CSEs (grade 1)

• Any other GCSEs, O levels or CSEs (any grades), Basic skills course

AS, A level or equivalent

• 2 or more A levels, 4 or more AS levels

• 1 A level, 2-3 AS levels

• 1 AS level

NVQ or equivalent

• NVQ level 3, BTEC National, OND or ONC, City and Guilds Advanced Craft

• NVQ level 2, BTEC General, City and Guilds Craft

• NVQ level 1

Other formal qualifications

• Qualifications achieved in England or Wales

• Qualifications achieved outside England and Wales

There was also an option at the end of this question for people to select "No formal qualifications".

Online, the order of the qualifications ran from NVQ to GCSE. Each question and the options after it were put on a separate screen with the option to tick "None of these apply" on each screen and a message that further questions would follow.

The results of the cognitive interviews showed that participants found the questions easy to answer. Those with qualifications achieved outside England and Wales found the questions more difficult to answer but understood that they needed to record equivalent qualifications using the examples provided as a guide.

The findings from the quantitative test confirmed the findings of the cognitive interviews. The findings showed that 92% of participants found it easy to answer the questions. In addition, 85% of those completing the paper version of the questions and 87% of those completing the online version of the questions were able to record their Highest Qualification. Those with qualifications from outside the UK were less likely to be able to record their qualifications than those with qualifications from the UK, although 70% still managed to record an equivalent qualification.

The main recommendations from this testing were to make the distinction between the qualifications questions and the option to mark "no formal qualifications" clearer by using "Or" to make them mutually exclusive and to improve the visibility of the "Tick all that apply" instruction on paper.

Based on the findings from these two rounds of testing (2018:39 and 2018:43) and space constraints, some final changes were made to the questions:

Given the space constraints on paper, the two options "Yes, in England and Wales" and "Yes, anywhere outside of England and Wales", which appeared under the question "Have you achieved any other qualifications?", were removed and replaced with one option "Other qualifications, equivalent unknown". The electronic questionnaire has both response options.

On paper, the section containing "Other qualifications, equivalent unknown" and "No qualifications" was separated from the other options by the word "Or" to show that this section is mutually exclusive from the other qualifications options. Online, the routing happens automatically, so the screen with these options will only be shown if the respondent has not selected any of the qualifications on the previous screens.

2019 Rehearsal findings

In the 2019 Rehearsal (2019:15), each question on the electronic questionnaire had the instruction "Include equivalent qualifications achieved anywhere outside England and Wales". Feedback showed that some people did not understand this instruction; some misinterpreted it to mean only qualifications achieved outside England and Wales should be included. Therefore, the instructions have been changed to "This could be equivalent qualifications achieved anywhere outside England and Wales".

There were also comments from respondents in the 2019 Rehearsal (2019:15) that suggested that some people misinterpreted or missed the question about degree-level or above qualifications on the electronic questionnaire.

Therefore, guidance about this question has been added in the form of an accordion that provides definitions of terms used. Guidance accordions have also been added to the NVQ, A level and GCSE questions. We present the text of each of these guidance accordions in Section 6: Questions recommended for Census 2021.

Welsh language question development

Between 2017 and 2018, an external agency with Welsh-speaking researchers was commissioned to undertake focus groups (2017:17) and a series of cognitive interviews (2017:18). In 2018, further cognitive interviews (2018:41) were undertaken by the same agency. The qualitative research tested public acceptability and comprehension of amended and newly designed census questions in Welsh. The questions were tested with people across Wales with varying dialects and Welsh language proficiencies.

To ensure questions adhere to Cymraeg Clir guidelines, some changes to the text or questions across the census questionnaires were translated by our contracted specialist Welsh language translation service provider. These changes were quality assured by the Welsh Language Census Question Assurance Group. This group includes Welsh language and policy experts from the Welsh Language Commissioner and the Welsh Government and was convened to give advice on the accuracy, clarity and acceptability of the language as well as other policy issues pertaining to the Welsh language and bilingual design.

The translations of the questions were positively received by Welsh speakers in three separate rounds of testing. However, two small changes to the Welsh language version of the questions were made as a result of this testing: colloquial translation of "Lefel A" (A level) was added alongside the formal translation – Lefel A (Safon Uwch) – and a colloquial translation of "Lefel AS" (AS level) was added alongside the formal one – Lefel AS (Safon UG).

The design of the questions recommended for Census 2021 was informed by the research and testing detailed in this report. The questions and response options for Census 2021 have now been finalised through the census secondary legislation.

The images that follow are from the electronic questionnaire. We have also published the paper questionnaires for Census 2021.

Guidance text and instructions are not part of the legislation, but we consider these to be finalised as well. However, it is possible that guidance text or instructions may change if there is enough evidence to support doing so.

As in previous censuses, only those aged 16 years or older will be asked about their qualifications.

Interstitial page

The qualifications questions begin with a short introduction that includes guidance on how to answer the questions.

In English, the text is “The next set of questions is about any qualifications you have ever achieved in England, Wales or worldwide, including equivalents, even if you are not using them now.”

In Welsh, the text is “Mae'r gyfres nesaf o gwestiynau yn ymwneud ag unrhyw gymwysterau rydych chi wedi’u hennill erioed yng Nghymru, Lloegr neu unrhyw le arall yn y byd, gan gynnwys cymwysterau cyfatebol, hyd yn oed os nad ydych yn eu defnyddio erbyn hyn.”

In the questionnaires for Wales, as with all other questions, instructions and guidance accordions, the list of home nations is written with Wales first, so the list is “Wales, England or worldwide” or “Nghymru, Lloegr neu unrhyw le arall yn y byd”.

Apprenticeships

Figure 1: Online question on apprenticeships in English and Welsh

Source: Office for National Statistics – Qualifications question development for Census 2021

Download this image Figure 1: Online question on apprenticeships in English and Welsh

.png (91.6 kB)

The questionnaires for Wales use the examples of trade, higher, foundation or modern apprenticeships.

Degree-level or above qualifications

Figure 2: Online question on degree-level or above qualifications in English and Welsh

Source: Office for National Statistics – Qualifications question development for Census 2021

Download this image Figure 2: Online question on degree-level or above qualifications in English and Welsh

.png (153.3 kB)

The accordion guidance text reads:

What we mean by “degree level or above”

This is any higher education qualification, achieved in the UK or another country, at level 4 or above.

Students are usually aged 18 years or over when they begin these qualifications through a college or university.

Examples include level 4 and 5 qualifications, bachelor’s degrees with or without honours, master’s degrees, PhD or other doctorates and professional qualifications, such as a PGCE or chartership.

NVQ qualifications

Figure 3: Online question on NVQ qualifications in English and Welsh

Source: Office for National Statistics – Qualifications question development for Census 2021

Download this image Figure 3: Online question on NVQ qualifications in English and Welsh

.png (210.6 kB)

The accordion guidance text reads:

What we mean by “NVQ”

This is a National Vocational Qualification. NVQs are competency and skills-based qualifications that can be achieved in school, college or at work.

If you have achieved similar qualifications, such as Scottish Vocational Qualifications or other vocational qualifications outside of the UK, choose the options you think are the closest match.

AS and A level qualifications

Figure 4: Online question on AS- and A-level qualifications in English and Welsh

Source: Office for National Statistics – Qualifications question development for Census 2021

Download this image Figure 4: Online question on AS- and A-level qualifications in English and Welsh

.png (209.4 kB)

The electronic questionnaires for Wales, in English and Welsh, include the response option “Bagloriaeth Cymru – Uwch” or “Advanced Welsh Baccalaureate”.

The accordion guidance text reads:

What we mean by “AS and A level”

These are advanced-level, subject-based qualifications that are often needed to get a place at university.

Students in England and Wales usually complete AS levels by the age of 17 years and A levels by the age of 18 years.

If you have achieved similar qualifications outside of England and Wales, choose the options you think are the closest match.

An International Baccalaureate diploma is equivalent to three A levels.

GCSE qualifications

Figure 5: Online question on GCSE qualifications in English and Welsh

Source: Office for National Statistics – Qualifications question development for Census 2021

Download this image Figure 5: Online question on GCSE qualifications in English and Welsh

.png (221.2 kB)

There are two more response options on the electronic questionnaires in Wales, in Welsh and English: “Bagloriaeth Cymru – Canolradd neu Genedlaethol” or “Intermediate or National Welsh Baccalaureate” and “Bagloriaeth Cymru – Sylfaen” or “Foundation Welsh Baccalaureate”.

The accordion guidance text reads:

What we mean by “GCSE”

This is a General Certificate of Secondary Education. GCSEs are subject based. Students in England and Wales usually complete GCSEs at school by the age of 16 years.

If you have achieved CSEs, O levels or other similar qualifications outside of England and Wales, choose the options you think are the closest match.

Question on any other or no qualifications

Figure 6: Online question on any other or no qualifications in English and Welsh

Source: Office for National Statistics – Qualifications question development for Census 2021

Download this image Figure 6: Online question on any other or no qualifications in English and Welsh

.png (124.1 kB)

Routing

The “any other or no qualifications” questions will only be shown if respondents have not selected any qualifications from the previous questions.

Question order

The qualifications question on the 2011 Census was placed between the health and employment questions. Respondents aged 15 years or under were routed around the questions on qualifications and employment. For Census 2021, some new questions have been added to the questionnaire. These include two voluntary questions on sexual orientation and gender identity and a question asking if the respondent has previously served in the armed forces. These new questions, as well as the qualifications question, are only asked to respondents aged 16 years or older; therefore, it made sense to keep all these questions in the same section of the questionnaire.

The qualifications question, which has a high level of public acceptability, was placed between these new voluntary questions and the armed forces question. This is because during testing (2018:27), we found that acceptability of the armed forces question improved when it was separated from the sexual orientation and gender identity questions. Therefore, the order of the questions is:

• sexual orientation

• gender identity

• qualifications

• armed forces service

• employment

On paper, this layout means the qualifications question starts on a new page, allowing respondents to view the instructions and the whole set of qualifications questions on the same page.

Question evaluation

The question design put forward in this report is based on extensive research and assessment using evaluation criteria that were set out in the publication The 2021 Census – Assessment of initial user requirements on content for England and Wales: Response to consultation (PDF, 796KB).

The evaluation considered the potential impact that including a topic on the census would have on data quality, public acceptability, respondent burden, financial concerns and questionnaire mode. The evaluations were used in conjunction with the user requirements criteria to steer the development of the census questions and questionnaire.

A topic that has been assessed as having a "High" potential for impact is closer to the threshold for exclusion from the census than a topic that has been assessed as having a "Low" potential for impact.

After completing the research and development phase, we evaluated the recommended questions against the same criteria using an updated tool that considers the type of evidence we have available and the Census 2021 context. A description of this updated evaluation tool is provided in the Question and questionnaire development overview for Census 2021.

All questions meet our thresholds to ensure reliable information will be collected in Census 2021.

Table 2 provides the updated evaluation scores for the qualifications questions. Potential for impact on criteria has remained "Low" for public acceptability and financial concerns. The potential for impact on the other criteria is lower than previously except for respondent burden and questionnaire mode for the other qualifications question. We have presented the evidence used to assess questions as having a "Medium" or "High" potential for impact on the evaluation criteria in Annex 3.

Question harmonisation

As in previous censuses, there will be separate censuses in Scotland and Northern Ireland. The questions for England and Wales have been developed through close collaboration with National Records of Scotland (NRS) and the Northern Ireland Statistics and Research Agency (NISRA), which are responsible for conducting the censuses in Scotland and Northern Ireland, respectively.

We recognise that each country has its own user and respondent needs; however, we aim for harmonisation of census questions and topics where possible to produce UK-wide statistics that are consistent and comparable.

On the NISRA census form, the order of the questions is:

• Degree level or above qualifications

• Any other qualifications

• Apprenticeships

The questions on "Any other qualifications" contains a list of other qualifications that is not separated into sections, as it is on Census 2021 in England and Wales.

The NRS census form has one question on qualifications with a list of possible qualifications to choose from. The list includes similar qualifications to those used in England and Wales on Census 2021.

The qualifications questions were developed for use in the context of Census 2021 in England and Wales, a mandatory household form. Therefore, it is possible that in different contexts, such as social surveys, a different approach may be more suitable.

The Government Statistical Service (GSS) proposes asking three questions to record level of education attainment and qualifications. However, the questions are aimed at recording the highest level of qualification and only record data at three levels: no qualifications, qualifications below degree level, and qualifications at degree level of above. Therefore, the output does not align with many stakeholders' need for more detailed information on qualifications.

In the Census 2021 topic consultation response, the Office for National Statistics (ONS) made clear commitments to the public. We committed to continuing to ask questions on the topic of qualifications. We also made commitments to:

• shorten and simplify the qualifications question to ease respondent burden

• investigate how data quality can be improved, drawing on the opportunities of a primarily online census

• work with stakeholders to understand the details of their requirements

• investigate the use of administrative data on educational qualifications

Simplifying and shortening the qualifications question

We simplified the qualifications question by:

• dividing the question up into shorter, less burdensome questions, instead of one long question containing a large amount of text

• simplifying the question; the questions only include the most common qualifications, and qualifications are grouped together in a logical way

• making apprenticeships and degree-level qualifications standalone questions

We investigated shortening the qualifications questions in February 2018, when we conducted a stakeholder survey. The survey included some shorter example questions; however, none of the shorter questions suggested met the main requirements of the stakeholders, so these questions were not taken forward to further testing.

Improving data quality and developing the electronic questionnaire

Our development process has aimed to improve the quality of data by making it easier for respondents to record their qualifications. We have followed best practices, including the Government Digital Service's Service Standard, when designing and developing the electronic questionnaire. We present further details of new functionality in the electronic questionnaire designed to improve the online experience in the Question and questionnaire development overview for Census 2021.

Working with stakeholders

We worked with stakeholders such as the Department for Business, Innovation and Skills (BIS), the Department for Education (DfE) and the Senedd (Welsh Parliament) during the development process by holding regular topic group meetings for the qualifications and education topic that included stakeholders; this allowed these stakeholders insight into the development process at each stage and to be involved in the process of developing the questions. We also worked with them in February 2018, when we conducted a stakeholder survey to allow stakeholders to provide detailed feedback on their requirements.

Investigating the use of administrative data

The ONS is exploring the use of administrative data on qualifications as a replacement for collecting such information in censuses and surveys; administrative data have the potential to provide more accurate information on qualifications achieved by individuals than self-reported data.

In October 2019, early research demonstrating the potential of administrative data to provide information on educational qualifications was published. This feasibility research showed that administrative data provided high-quality information on the highest level of qualification in 2011 for individuals aged 16 to 25 years who studied in government-funded education in England. This offers insight into a large proportion of first-time entrants to the labour market and consequently an understanding of whether this group is equipped with the skills to meet market demands.

Further research is required to provide information on qualifications for all persons aged 16 years and over in England and Wales; therefore, surveys including the census remain the best way to collect this information to meet users' needs at this point in time.

Although our early research is encouraging, the ONS needs to further demonstrate that the dataset can provide statistical information on educational attainment to the level of statistical quality that will meet users' needs.

References to tests take the form (Year: Test number). "Year" refers to the calendar year the test was undertaken in and the test number is the position of the test within the year considering all testing that took place in that year. For example, the fifth test conducted in 2017 would be referenced as (2017:5).

A full description of each of these items can be found in the Summary of testing for Census 2021.

Evaluation of apprenticeships question

Potential for impact on data quality, public acceptability, respondent burden and financial concerns were assessed as "Low".

Potential for impact on questionnaire mode: "Medium"

This question has different instructions and layout online compared to the paper questionnaire as well as radio buttons that prevent more than one answer being given.

Evaluation of degree question

Potential for impact on public acceptability and financial concerns were assessed as "Low".

Potential for impact on data quality: "Medium"

This question is new for Census 2021, has the potential for recall bias and asks for information that cannot be directly observed. In addition, feedback from the 2019 Rehearsal (2019:15) showed that some participants misunderstood or missed this question. For Census 2021, we have implemented a few changes, including adding more examples and accordion guidance to help respondents answer this question.

Potential for impact on respondent burden: "Medium"

Feedback from the 2019 Rehearsal (2019:15) showed that some participants found this question difficult to answer. In addition, this is a new question in Census 2021 and asks for information that cannot be directly observed, which can make it difficult to answer on behalf of another person.

Potential for impact on questionnaire mode: "Medium"

This question has different instructions and layout online compared to the paper questionnaire as well as radio buttons that prevent more than one answer being given.

Evaluation of other qualifications question

Potential for impact on public acceptability and financial concerns were assessed as "Low".

Potential for impact on data quality: "Medium"

This question is new for Census 2021, has the potential for recall bias and asks for information that cannot be directly observed. Analysis of data from the 2019 Rehearsal (2019:15) showed that a higher than expected percentage of participants used the "Previous" button when answering these questions, suggesting that they were unsure of their answers.

Potential for impact on respondent burden: "High"

The question stem, instructions and response options are long, so respondents have a lot of text to read and understand when answering each question. This question can also be difficult to answer on behalf of another person and asks respondents to recall past events that may have happened a long time ago.

Analysis of data from the 2019 Rehearsal (2019:15) showed that a higher than expected percentage of participants used the "Previous" button when answering these questions and the average time spent on the screens was also longer than expected. These results suggest respondents had difficulty answering the questions.

Potential for impact on questionnaire mode: "High"

On paper, these questions are all presented as one question, split into sections. On the electronic questionnaire, it is split into multiple questions across four screens, with different instructions and layout to the paper questionnaire. Mutually exclusive response options are used online, and there are more response options available online than on paper.