1. Disclaimer

These Research Outputs are not official statistics relating to the labour market. They are published as outputs from research into testing the effects of attrition on response rates across three waves of a survey. This will help development of an alternative prototype survey, known as the Labour Market Survey (LMS), to the one currently used in the production of labour market statistics, the Labour Force Survey (LFS).

It is important that the information and research presented here is read alongside the accompanying technical report to aid interpretation and to avoid misunderstanding. These Research Outputs must not be reproduced without this disclaimer and warning note.

Nôl i'r tabl cynnwys

2. Overview

Between April 2019 and November 2019, the Office for National Statistics (ONS) conducted a large-scale attrition test of the Labour Market Survey (LMS) using an online-only approach to data collection. The purpose of the test was to measure response and attrition rates across three waves of data collection. This test was the next iteration in a series of tests that formed an important part of an on-going research programme that is being conducted as part of the ONS Census and Data Collection Transformation Programme.

For wave 1 of the LMS Attrition Test, 50,000 private households were sampled across Great Britain using AddressBase as a sampling frame. Data collection was online only and the collection period at each wave was three weeks. Households that responded either fully or partially (partial being at least one person in the household had fully completed their individual section of the survey) were sampled again at wave 2, which took place three months after the start of wave 1. The same principle applied at wave 3, which took place three months after wave 2 and comprised of fully or partially completing households. Non-contact households and refusals at wave 1 were not re-issued to wave 2, nor were non-contacts and refusals at wave 2 reissued to wave 3.

There were several experimental conditions that were implemented at wave 2 to test the effect of incentivisation and reduced communications on attrition rates. These are detailed in the technical report, along with the achieved response rates at each wave. The socio-demographic characteristics detailed in this report cover respondents across all experimental conditions, as the design of the test was to look at the differences in response rates between conditions rather than the composition of each responding sample.

For a comprehensive summary of the context and design of the LMS Attrition Test, please see the accompanying technical report.

This report provides a summary of the socio-demographic characteristics of the responding sample across three subsequent waves of the LMS Attrition Test. We make comparisons with the responding sample for the Labour Force Survey (LFS) over a comparable data collection period. Please note that the LFS uses face-to-face interviewing as the primary mode for data collection at wave 1, with subsequent waves collected by telephone or face-to-face. Further comparisons are also made with the results from the LMS Statistical Test to explore differences in characteristics using a mixed-mode approach to data collection (online by default followed up with face-to-face interviewing) from a single wave perspective.

All data presented throughout this report are also measured against our 2019 mid-year population estimates where available. No statistical tests have been carried out on these data as this was not part of the scope of this test. The results presented will require further consideration and will form the basis of further evaluation. Along with the initial evidence provided in the other reports, this evaluation will inform the future design and testing of the LMS.

This report should also be considered alongside the results and evidence presented in the technical report, which details how the LMS Attrition Test can be compared to the equivalent LFS collection. The differences in the socio-demographic characteristics of the responding sample are indicative of potential differences based on the two survey designs.

Nôl i'r tabl cynnwys

3. Datasets used for comparisons

Fourteen datasets have been used in this report for the purpose of comparing the socio-demographic characteristics of responding individuals from the Labour Market Survey (LMS) Attrition Test, the Labour Force Survey (LFS) and the LMS Statistical Test. The datasets are comprised of individual and household level types and are explored in more detail below. All figures presented in this report are produced from unweighted data. This allows for the report to highlight any areas of notable differences from these raw unweighted counts and percentages, and to suggest avenues for further research and consideration regarding the survey design of the LMS.

Labour Market Survey Attrition Test datasets

The full unweighted datasets from the LMS Attrition Test, covering the period of April 2019 to November 2019, are used for the LMS Attrition Test analysis presented in this report. Analysis has been conducted on an individual level for Tables 1 to 6, with Tables 7, 8a and 8b having been conducted on a household level. As an online only test, the datasets contain online responses only. Each dataset is described in more detail below.

Wave 1 datasets

The full unweighted datasets for wave 1 of the LMS Attrition Test, covering the period from 22 April 2019 to 19 May 2019, are used for the analysis presented in this report. The individual level dataset contains 28,460 responding individuals (24,021 adult responses) from across Great Britain, including both individual and proxy responses. The household level dataset contains 13,473 responding households.  

Wave 2 datasets

The full unweighted datasets for wave 2 of the LMS Attrition Test, covering the period from 22 July 2019 to 18 August 2019, are used for the analysis presented in this report. The individual level dataset contains 15,469 responding individuals (13,563 adult responses) from across Great Britain, including both individual and proxy responses. The household level dataset contains 7,543 responding households.

Wave 3 datasets

The full unweighted datasets for wave 3 of the LMS Attrition Test, covering the period from 21 October 2019 to 17 November 2019, are used for the analysis presented in this report. The individual level dataset contains 10,009 responding individuals (9,022 adult responses) from across Great Britain, including both individual and proxy responses. The household level dataset contains 5,065 responding households.

Labour Force Survey comparison datasets

The full unweighted bespoke datasets for the LFS, covering the same period of April 2019 to November 2019, are used for all LFS analysis presented in this report. Analysis has been conducted on an individual level for Tables 1 to 6, with Tables 7, 8a and 8b having been conducted on a household level. The LFS is a mixed-mode survey, predominantly conducted through face-to-face interviews with telephone follow up. The datasets therefore contain both forms of responses.

The LFS comparative datasets used in this report differ from the full LFS datasets used to produce published estimates in that they only include wave 1 responses and exclude all individual responses achieved for the longitudinal component of the full quarterly LFS dataset (waves 2 to 5). They also exclude all responses achieved in Northern Ireland and north of the Caledonian Canal, to correspond to the sample area used for the LMS Attrition Test and LMS Statistical Test.

Wave 1 datasets

The full comparative unweighted datasets for wave 1 of the LFS cover the same period as wave 1 of the LMS Attrition Test and are used for the analysis presented in this report. The individual level dataset contains 5,969 responding individuals (4,777 adult responses) from across Great Britain, including both individual and proxy responses. The household level dataset contains 2,615 responding households.

Wave 2 datasets

The full comparative unweighted datasets for wave 2 of the LFS cover the same period as wave 2 of the LMS Attrition Test and are used for the analysis presented in this report. The individual level dataset contains 5,523 responding individuals (4,322 adult responses) from across Great Britain, including both individual and proxy responses. The household level dataset contains 2,279 responding households.

Wave 3 datasets

The full comparative unweighted datasets for wave 3 of the LFS cover the same period as wave 3 of the LMS Attrition Test and are used for the analysis presented in this report. The individual level dataset contains 4,359 responding individuals (3,451 adult responses) from across Great Britain, including both individual and proxy responses. The household level dataset contains 1,850 responding households.

Labour Market Survey Statistical Test dataset

The full unweighted datasets for the LMS Statistical Test do not cover the same period as the LMS Attrition Test or LFS datasets. It is presented in this report to explore differences in characteristics using a mixed-mode approach to data collection (online by default followed up with face-to-face interviewing) from a single wave perspective only. The individual level dataset contains 17,237 responding individuals (14,255 adult responses) from across Great Britain, including both individual and proxy responses. The household level dataset contains 7,718 responding households. As a mixed-mode survey, the datasets contain both online and face-to-face responses.

The mid-year population estimates 2019

Mid-year estimates (MYEs) are the official source of population sizes in-between censuses, covering populations of local authorities, counties, regions and countries of the UK by age and sex. The estimates use the census definition of people who are "usually resident" in the UK for 12 months, excluding short-term migrants, and counting students at their term-time addresses. The estimates roll forward the population found by the previous census one year at a time by accounting for births, deaths, international migration and internal migration. To accomplish this multiple registration, survey and administrative data sources are used including the General Register Office (GRO), the International Passenger Survey (IPS), the Higher Education Statistics Agency (HESA), the National Health Service Central Register (NHSCR) and the Ministry of Justice (MoJ). Comparable estimates for Scotland and Northern Ireland are produced by National Records of Scotland (NRS) and the Northern Ireland Statistics and Research Agency (NISRA) respectively.

The 2019 population projections for Great Britain are used in this report; it was not possible to exclude north of the Caledonian Canal from these projections, but the impact of the inclusion is likely to be negligible as it comprises a very small proportion of the overall population projection.

Nôl i'r tabl cynnwys

4. Sex and age bands

Sex

Table 1 presents the proportion of the responding samples by sex for each wave of the Labour Market Survey (LMS) Attrition Test and Labour Force Survey (LFS) and provides a comparison against the Statistical Test and the mid-year estimates (MYEs). The figures achieved at wave 1 for the LMS Attrition Test are comparable with all waves of the LFS, Statistical Test and mid-year estimates (MYEs), with a higher response received from females. Interestingly, the opposite was found at wave 2 and wave 3 of the LMS Attrition Test, with males being more likely to participate than females. For the LMS Attrition Test, there was an increase in the proportion of responses that were from males across waves (3.2% increase between waves 1 and 2, 3.0% increase between waves 2 and 3). However, there was an inverse relationship by female respondents, which decreased across waves (3.2% decrease between waves 1 and 2, 3.0% decrease between waves 2 and 3.) These findings differ to that found on the LFS where the relationship across male and female respondents remained relatively static across waves, with female respondents consistently more likely to participate.

The collection of a respondent’s sex is a mandatory response variable on the LFS, with interviewers required to ask or record this for each individual. There are therefore no missing values in the LFS data. No variables were mandatory for completion on the LMS Attrition Test and Statistical Test and it was possible for respondents to bypass questions or to leave their answers blank. However, upon examination of the Attrition Test data, there were no missing values for sex across all waves.

Age

Table 2 presents the proportional distribution by age band for the responding samples of the LMS Attrition Test and LFS, comparing these with the distributions from the Statistical Test and MYEs. Compared with the LFS, Statistical Test and MYEs, respondents aged 55 to 64 years were over-represented in the LMS Attrition Test across all waves. Respondents aged 65 years and over were the largest responding group and were also over-represented across all waves of the LMS Attrition Test, which is comparable with wave 1 of the LFS, Statistical Test and MYEs. Interestingly, for this age group there is a significant increase in the proportion of response across all waves for the Attrition Test (7.9% increase between waves 1 and 2, 6.9% increase between waves 2 and 3). A lower number of respondents aged 15 years and under were observed on the LMS Attrition Test across all waves compared with the LFS. The group with the lowest proportion of responders across all waves of the LMS Attrition Test were those aged 16 to 24 years, which is a consistent finding across all measures.

Age is another variable that is mandatory for completion by interviewers on the LFS, resulting in no instances of missing data. There are various possible reasons for missing values on the LMS Attrition Test, such as respondents having concerns about confidentiality or being unable to recall this information. Please see footnote detailed above for the percentage of missing values for the Attrition Test.

Age by sex

Tables 3a and 3b present the responding samples for the LMS Attrition Test and LFS, categorised by age band and sex. For the LMS Attrition Test, both males and females aged 16 to 24 years and 25 to 34 years were under-represented. Specifically, for males aged 16 to 24 years, the LMS Attrition Test achieved a lower proportion of response in comparison with the LFS and were the lowest responding demographic across all measures. Their response decreased across waves (0.6% decrease between waves 1 and 2, 1.1% decrease between waves 2 and 3) on the LMS Attrition Test whereas their response pattern fluctuated across the LFS (0.3% increase between waves 1 and 2, 0.8% decrease between waves 2 and 3). In addition, females aged 16 to 24 years were consistently under-represented for both the LMS Attrition Test and LFS at wave 1. However, the LMS Attrition Test observed a decrease in proportion of response across waves (1.4% decrease between waves 1 and 2, 1.3% decrease between waves 2 and 3) whereas the LFS did not. This could be attributed to the difference in modal type across the LMS Attrition Test and LFS.

The proportion of male responders aged 65 years and over on the LMS Attrition Test was higher than female responders across all waves, whereas this pattern was not observed on the LFS. Interestingly, males aged 65 years and over had the highest increase in proportion of response across all waves of the LMS Attrition Test (8.5% increase between waves 1 and 2, 6.5% increase between waves 2 and 3) with nearly half of the male responding sample aged 65 years and over by Wave 3. A similar pattern was observed in females aged 65 years and over for the LMS Attrition Test (6.5% increase between waves 1 and 2, 6.2% increase between waves 2 and 3). Overall, the effect of attrition creates more of a skewed sample distribution by age for males than it does for females on an online-only survey.

Nôl i'r tabl cynnwys

5. Country of birth, ethnicity and nationality

Country of birth

Table 4 presents the responding samples for the Labour Market Survey (LMS) Attrition Test and Labour Force Survey (LFS) by country of birth. The majority of the responding samples for all measures stated they were born in the UK, with the LMS Attrition Test achieving a higher proportion of response in comparison with the LFS. wave 1 of the LMS Attrition Test achieved a UK response of 95.6% in comparison with 86.1% on the LFS, wave 2 achieved a UK response of 96.8% compared with 85.9% and wave 3 achieved a UK response of 97.4% compared with 86.5%. The LFS achieved a higher proportion of response from responders born in European countries and those from the Other category (countries outside the UK and EU) in comparison with the LMS Attrition Test across all waves.

However, it is important to note that the LMS Attrition Test did not include a coding frame for the Other option for the variables country of birth, ethnicity and nationality. These questions were not mandatory to answer, and responders had the option to state their country of birth in a free-text field. By not using a coding frame it was possible to examine the quality of the free text entries by online respondents and code these to specific countries during the data cleaning process. On the LFS, interviewers code a respondent's country of birth during the interview, based on a coding frame. A possible reason for the differences observed for the country of birth characteristic across the LMS Attrition Test and LFS may be due to the difference in mode between the two measures; the presence of an interviewer on the LFS helps mitigate overall non-response to the survey and help overcome any language barriers when completing the survey.

Ethnicity

Table 5 presents the proportion of the responding samples for the LMS Attrition Test and LFS by their reported ethnicity. The LMS Attrition Test observed a relatively similar responding sample at wave 1 for the White British group compared with all waves of the LFS. However, a difference was observed across waves between the two measures. The proportion of 'White British' response for the LMS Attrition Test increased across all waves (3.3% increase between waves 1 and 2, 2.1% increase between waves 2 and 3). However, the responding sample remained static for the LFS across waves, which was similar to the response rate observed on the Statistical Test. The proportion of respondents from ethnic minority groups was marginally higher across the majority of waves on the LFS compared to the LMS Attrition Test. Similarly, the LMS Attrition Test also had a lower number of responders from the Black/African/Caribbean/Black British group across all waves compared with the LFS.

Nationality

Table 6 presents the responding samples for the LMS Attrition Test and LFS by their stated nationality. The vast majority of respondents reported their nationality as UK, British, with the LMS Attrition Test achieving a higher proportion of response in comparison with the LFS. wave 1 of the LMS Attrition Test achieved a 93.9% proportion of response for UK, British in comparison with 90.6% on the LFS. wave 2 achieved a 95.0% proportion of response compared with 90.2% and wave 3 achieved a 96.0% proportion of response compared with 90.6%. However, a difference was observed across waves between the two measures; the UK British proportion of response for the LMS Attrition Test increased across all waves (1.1% increase between waves 1 and 2, 1.0% increase between waves 2 and 3). However, the responding sample remained static for the LFS across waves, which was similar to the response profile observed on the Statistical Test. The LFS achieved a higher proportion of response from responders with a nationality representing one of the EUA8 nationalities and the Other category (nationalities outside the UK and EU) in comparison with the LMS Attrition Test across all waves. Again, when selecting the Other option, responders had the option to report their nationality in a free-text field. By not using a coding frame, it was possible to examine the quality of the free text entries by online respondents and to code these to specific countries during the data cleaning process.

Nôl i'r tabl cynnwys

6. Household size

The number of eligible people in each responding household and the mean household size for the Labour Market Survey (LMS) Attrition Test and Labour Force Survey (LFS) is shown in Table 7. The eligibility of individuals for inclusion in the household for both the LMS Attrition Test and LFS was defined as any adults, children or babies who define the sampled address as being their main residence, even if they were currently away for a continuous period of up to six months. The LFS permits up to 16 people to be added to the household, whereas the LMS Attrition Test allowed a maximum of 10. This was a tactical decision for the purposes of the test to avoid potential performance issues with the online data collection instrument.

There was little variation between the results from the LMS Attrition Test and LFS across all waves, demonstrating that for this characteristic there was consistency between the two surveys, despite the differences in the survey designs (detailed in the technical report). This is also evident in Table 7 which presents the mean number of eligible individuals in a household, which was 2.3 people for both the LMS Attrition Test and the LFS at wave 1; 2.1 people (LMS Attrition Test) and 2.4 people (LFS) at wave 2; and 2.0 (LMS Attrition Test) and 2.4 (LFS) at wave 3. Table 7 also demonstrates that there was little variation between the results from the LMS Attrition Test, LFS and Statistical Test - which was only one wave of data collection.

Nôl i'r tabl cynnwys

7. Index of multiple deprivation

The Index of Multiple Deprivation (IMD) is a relative measure of deprivation across small areas based on seven different domains of deprivation: income, employment, education, health and disability, crime, barriers to housing and services and living environment. Various indicators are used to measure deprivation in each domain, using information such as school performance or access to local hospitals. Each devolved government produces their own IMD, updating it every four years. The IMDs used in this analysis are the English indices of deprivation 2019, the Welsh Index of Multiple Deprivation 2019 and the Scottish Index of Multiple Deprivation 2016. This report presents response data categorised by IMD deciles, with Decile 1 representing the most deprived areas and Decile 10 representing the least deprived.

Table 8a presents the distribution of responding households for the Labour Market Survey (LMS) Attrition Test and Labour Force Survey (LFS) by IMD deciles across three waves of data collection. The LFS has a relatively even distribution of response for all IMD deciles across all waves, ranging from 9.0% (Decile 2) to 10.9% (Decile 7) at wave 1; 9.4% (Decile 9) to 10.8% (Decile 3) at wave 2; and 8.4% (Decile 1) to 11.0% (Decile 10) at wave 3. The LMS Attrition Test has a more varied response distribution, ranging from 6.2% (Decile 1) to 12.8% (Decile 10) at wave 1; 5.6% (Decile 1) to 13.9% (Decile 10) at wave 2; and 8.4% (Decile 1) to 11.0% (Decile 10) at wave 3. Overall, the relationship between response and IMD decile across the waves for the LMS Attrition Test was linear, with the proportion of response increasing as areas of deprivation decreased. Additionally, this distribution becomes more pronounced across waves, with the proportion of response further decreasing for IMD Decile 1 to 5 across time. In contrast, there was a more even distribution of response across IMD decile for all waves on the LFS.

Table 8b categorises the IMD deciles into areas of higher deprivation (Deciles 1 to 5) and areas of lower deprivation (Deciles 6 to 10). The distribution of response to the LFS is evenly split across areas of deprivation and consistent across waves. For wave 1 of the LFS, a response of 48.6% was recorded in areas of higher deprivation and 51.4% in areas of lower deprivation; wave 2 achieved a response of 50.0% in both areas of deprivation; and wave 3 achieved a response of 48.8% response in areas of higher deprivation and 51.2% in areas of lower deprivation. The distribution of response for the LMS Attrition Test is consistent across waves but less evenly split across areas of deprivation, with less response in areas of higher deprivation. wave 1 achieved a response of 40.0% in areas of higher deprivation and 60.0% in areas of lower deprivation; wave 2 achieved a response of 37.4% in areas of higher deprivation and 62.6% in areas of lower deprivation; and wave 3 achieved a response of 36.1% in areas of higher deprivation and 63.9% in areas of lower deprivation. Additionally, this distribution becomes more pronounced across waves, with the proportion of response further declining for those areas of higher deprivation across time. These results imply that the LMS Attrition Test achieved a less representative responding sample by deprivation indicator than the LFS. This difference in distribution of response across the LMS Attrition Test and the LFS could be attributed to a difference in mode, with the LFS able to achieve a more even distribution of response across areas of higher and lower deprivation due to the face-to-face interview nature of the data collection, whereas the LMS Attrition Test was offered online only.

The IMD of a property was not factored into the sampling process, and therefore the sampled distribution of addresses by IMD is a by-product of the process. There is further information on the sampling process, in the technical report.

Nôl i'r tabl cynnwys

8. Conclusion

There were some similarities found between household characteristics on the LMS Attrition Test and LFS - for example, household size and sex at wave 1. However, there were notable differences observed across all other characteristics. Specifically, on the LFS the relationship across male and female respondents remained relatively static across waves, with female respondents more likely to participate. However, this pattern was not observed across the waves on the LMS Attrition Test - with a lower female response at wave 2 and wave 3. The response rate for females also decreased across waves (3.2% decrease between waves 1 and 2.3% decrease between waves 2 and 3), suggesting females were less likely to further participate in the LMS Attrition Test across time.

The LMS Attrition Test had a higher responding sample for demographics such as: country of birth, ethnicity, and nationality, resulting in an over-representation of the UK, White British and UK, British categories respectively. For the country of birth characteristic, the LMS Attrition Test observed a lower response for the Other category in comparison with the LFS, which also declined across waves, whereas on the LFS it remained consistent across waves. Similarly, for ethnicity, the White British response on the LMS Attrition Test increased across all waves whereas the responding sample remained static across waves on the LFS. Comparatively, all other categories for ethnicity declined. Again, these findings may be attributed to the difference in mode between the surveys.

Regarding the age distribution of the responding sample, respondents aged 55 years and over were over-represented in the LMS Attrition Test compared with both the LFS and mid-year estimates (MYEs). Interestingly, there was a pronounced increase in response across all waves of the LMS Attrition Test for the 65 years and over age group when compared with the LFS - with a larger proportion of response from males in comparison with females. This age group also had the highest proportion of response for the LMS Attrition Test, which differs to the LFS and MYEs, who found the highest proportion of response from the 15 years and younger age group. Considering the modal differences between the two surveys this is an interesting finding, as the 15 years and younger age group would have had their responses provided by proxy for both measures. Most importantly, the data suggests that the effect of attrition creates more of a skewed sample distribution by age for male than it does for females on an online-only survey.

Some interesting results were also identified when analysing the responding sample using the Index of Multiple Deprivation (IMD). Across all waves for the LMS Attrition Test, the relationship between response and IMD decile was linear, with response increasing as areas of deprivation decreased. This contrasts with the LFS were there was a more even distribution of response across IMD decile for all waves. This pattern is also evident when categorising the responding sample into areas of high and low deprivation (see Table 8b), with the LMS Attrition Test observing less response in areas of high deprivation across all waves when compared with the LFS. It is important to note that the distribution of the Attrition Test became more pronounced across waves, with the proportion of response further declining for those areas of higher deprivation, resulting in a skewed sample profile across time.

These findings provide strong support for the need of an interviewer mode on each wave of the future Labour Market Survey (LMS) to avoid skewing the sample profile across time. In addition, the findings highlight several areas for exploration in future research and testing cycles, with the aim to further the understanding of any identified differences and feed into the future re-design of the LMS, where appropriate. To gather a more complete picture of the characteristic profile of the LMS responding sample, further work is planned to explore the effects of attrition on the characteristic profile of the LMS Beta Survey which is currently live in the field.

Nôl i'r tabl cynnwys

Manylion cyswllt ar gyfer y Methodoleg

Colin Beavan-Seymour
lms.transformation@ons.gov.uk
Ffôn: +44 (0) 1633 455536