Unpaid Carers Scoping Study, feasibility of a new survey, England: May 2024

• There is a need for more detailed information on unpaid carers that includes demographic information, the type of care being provided, the amount of time spent caring and the services that are used; this is important for understanding disparities and developing policy.

• A new survey could allow for a more representative view of the unpaid caring population, while capturing information about their caring experiences; the sampling method selected would have a substantial impact on how representative the survey is of the target population.

• A new survey would need a sample of around 45,000 unpaid carers to enable meaningful conclusions to be drawn at a local-authority level, which is important for policy development.

• A new survey should have an online-first approach, as paper surveys are resource intensive, and may not reach all carers; having paper surveys on request would enable inclusion of those who cannot access an online version.

• A longitudinal survey design could be considered; this would enable the collection of data over time, allowing for identification of changes over time to produce a richer assessment of the unpaid care sector.

• A pilot survey to test multiple sampling methods, including the possibility of a blended approach, is recommended; this would ensure any future survey designs are proven to work in the context of the unpaid care population, before the implementation of a full-scale survey.

Care data matters

In December 2023, the Department for Health and Social Care (DHSC) released the final version of their Care data matters: a roadmap for better social care data strategy, which explains the importance of data in ensuring people get the right care. The ambition set out is to increase the quality and quantity of social care data available.

In the case of unpaid carers, the report highlighted the need for better information on unpaid carers, including the type of care they provide, the amount of time spent caring, and the services they use. Improving demographic knowledge of the population, particularly around protected characteristics and health and wellbeing, is essential to understanding disparities in this group. This is crucial for policymaking and ensuring that unpaid carers and the people they care for get the required support.

Currently, NHS England run an Adult Social Care Survey (ASCS) and the Survey of Adult Carers in England (SACE). This scoping study has been completed to identify different options for undertaking a new survey, including how SACE could be incorporated.

Survey of Adult Carers in England

A discovery report (ASCS & SACE Discovery Report, PDF, 1,198KB) was completed in 2022. This discovery study explored how the Survey of Adult Carers in England (SACE) could be redesigned to become more representative of the unpaid carer population while capturing information about their experiences as carers.

The SACE is currently the largest data source about outcomes and experiences of unpaid carers receiving support from their local authority (LA). Its main purpose is to provide local-level data about support that can be used at a national level to contribute to measures in the Adult Social Care Outcome Framework (ASCOF).

SACE currently uses targeted random sampling and is a self-completion paper survey, delivered biennially. To be eligible for SACE, unpaid carers must meet these three criteria:

• aged 18 years or over

• caring for someone aged 18 years or older

• present in the final four rows of the Short- and Long-Term Support (SaLT) return (NHS Digital's measure LTS003, which identifies carers who have received support from the LA within the previous 12 months)

Currently, each LA follows NHS guidance to draw their own sample from the SaLT return, administering the survey themselves. At the time of the most recent SACE in 2021 to 2022, there were 152 councils with adult social services responsibilities (CASSRs), 149 of which took part. The sampling does not capture anyone outside of the SaLT return, so only carers who have received support from their LA in the 12-month period are eligible to take part.

The discovery report considered ways that the survey could be made more accessible and representative. Thought was given to the mode of the survey, with a recommendation to consider a push to web approach to maximise time and cost savings for LAs while broadening completion options for unpaid carers.

Improving participation from unpaid carers was identified as essential to improving general representativeness of SACE. Several methods were considered including improving signposting for different versions of the survey such as translated versions, as well as engagement with underrepresented groups to encourage participation.

As well as suggestions for maximising completions from unpaid carers invited to SACE, it was suggested that broadening the scope of SACE to include more unpaid carer groups, such as self-funders and those providing short-term care, was favourable among stakeholders. Two main challenges with this were identified:

• implications for LAs administering the survey because of increased cost and resource requirements for a larger sample size

• challenges with sourcing contact details of other unpaid carers, making it difficult to identify an appropriate sample

The findings of the discovery report highlighted important areas for improvement of SACE, informing the focus of this scoping study.

The scoping study focused predominantly on approaches to improve representation in future survey designs and delivery. These have been considered when assessing possible sampling frames and their viability.

Identifying and targeting underrepresented groups

Underrepresented groups refer to those people who are insufficiently or inadequately represented in the survey data, this generally includes those where the overall size is small, identification is difficult, or where there are incomplete or non-existent sampling frames.  

Underrepresented groups often reflect different experiences that need to be recognised and understood to provide full coverage and insight into a population of interest. Differences in response rates from different groups can also affect the representativeness of the survey. 

An in-depth review of underrepresented groups included a combination of desk research, stakeholder engagement and feedback from local authorities. Over 20 underrepresented groups in the data were identified. Ensuring inclusivity in survey design and collection is multidimensional with varying needs across and within groups, which makes capturing them all particularly challenging.

Importantly, this still may not represent all groups who tend to be underrepresented in the data. Furthermore, there may be differences across areas and surveys in the level of underrepresentation seen. In the SACE more specifically, the sampling method may mean the sample is not completely representative of the caring population, as the sample frame only includes unpaid carers who have contacted their local authority for support.

Approaches that have been used to target underrepresented groups within research by the Office for National Statistics (ONS), as well as other organisations, have been explored as part of the scoping study. The considered approaches include:

• broadening the range of methods and creating new approaches to understanding experiences

• ensuring outputs are accessible

• offering different methods of engagement, multiple modes and design formats, to increase inclusion and to maximise participation

• engaging with a range of experts such as academics, policy experts and third-sector organisations

• using unpaid carers as "experts by experience" to support development, ensuring materials and overall design decisions are fit for purpose

• signposting different formats of the survey

• ensuring that online materials meet accessibility standards

Self-identification as an unpaid carer

A challenge in capturing information highlighted by stakeholders is self-identification, where those providing unpaid care by definition may not recognise themselves as doing so. This can be a substantial barrier to completing a survey on this topic.

Through engagement with experts, possible methods of increasing the likelihood of self-identification and reducing related barriers to taking part have been considered. These could be applied to any survey design to increase the number of carers captured:

• avoiding the terms "carer" and "unpaid carer", as these could make people believe they are not eligible to complete the survey if they do not identify with this term

• asking respondents if they complete any tasks (through selection of multiple choices) that would meet the definition of unpaid caring

• using showcards that provide examples of caring activities

• using the Government Analysis Function's unpaid care harmonised standard question, as this will be comparable with the 2021 Census as well as other data sources (it will therefore also be familiar to many people)

Throughout this scoping study, we have prioritised the sampling for a new survey to ensure that it could achieve a representative sample size. 

The Survey of Adult Carers in England (SACE) is designed so that the 95% confidence interval around an estimate of 50% should be no more than plus or minus 5 percentage points​. An estimate of 50% refers to the random probability of someone choosing between two answer options. A 95% confidence interval is a range within which the true population would fall for 95% of the times the sample survey was repeated. It is a standard way of expressing the statistical accuracy of a survey-based estimate. If an estimate has a high error level, the corresponding confidence interval will be very wide.

Future surveys should aim to at least match the accuracy of SACE. To achieve a plus or minus 5% confidence interval at a local-authority level, on average 300 achieved responses per local authority would be required. This means that the national overall achieved sample would be around 45,000, with a margin of error of plus or minus 0.5%. 

Sample designs involving a known carer population

Follow-on survey sampling

We explored the option of using existing surveys as a follow-on sample for a new survey of carers. This involved identifying surveys that captured whether someone provides unpaid care, and the number of those who consented to being recontacted for another survey.

Engagement showed that a sample from current sources would not be viable to form a sole sample source because of small numbers and limited representativeness of identified unpaid carers compared with the sample size required. 

Targeted sampling

Targeted sampling would use an existing database of carers as a sample frame. The SACE sample design, using the Short- and Long-Term Support (SaLT) return, is an example of this. This involves taking a random sample of a group of known carers and inviting them to complete a survey. This reduces the likelihood of potential respondents being ineligible, saving costs while increasing response rates. While targeted sampling would be favourable because of the reduced risk of ineligibility, there is not currently an accessible, representative sample frame of known unpaid carers. If this were to change, it should be explored as an option to capture unpaid carers.

Sample designs without a known unpaid carer population

Other sampling designs, where a source of known unpaid carers is not available, have been assessed to determine whether they would be viable options for a future survey.

For each of the following sample designs, an assumed response rate (RR) of 33% has been used based on current SACE response rates.

Random sampling

Random sampling is a type of probability sampling where everyone in the entire target population has an equal chance of being selected, meaning that the issued sample should be representative of the entire population. While the representativeness of random sampling is good, it does not allow for targeting typically underrepresented groups to boost their sample and does not consider groups that may have higher proportions of non-response. Random sampling can also be time, cost and resource intensive, depending on the eligibility criteria and the size of the issued sample.

Unpaid carers only account for 8.8% of the usual resident population aged five years and over in England (Census 2021), as shown in our Unpaid care by age, sex and deprivation dataset, so a random sample of the whole population may not be feasible. The scale of the survey would need to be large to meet the desired sample size of 45,000 completions. As less than 1 in 10 respondents would be expected to be eligible, around 450,000 people would need to be invited. Furthermore, to account for non-response (with an assumed 33% response rate), the issued sample would need to be around 1.4 million. Random sampling would be extremely resource intensive because of costs associated with inviting ineligible people.

Stratified random sampling

Stratified sampling involves dividing the population into non-overlapping groups (strata) and taking a random sample independently in every group.

If individuals within a group have similar characteristics, stratification can reduce standard errors. For example, if a survey is stratified by region, the number sampled in each region is "fixed" and not subject to random variability. This reduces sampling variability as individuals within a group are similar to each other and because it would be impossible to randomly sample more or fewer cases than expected in any given region, regional distributions are proportionate. Stratified sampling is more cost effective than a true random sample because smaller issued sample sizes are required. Analysis of Census 2021 results can be used to determine the variables suitable for stratification in a new survey.

Self-select sampling

A self-select approach has been used to capture unpaid carers in other surveys conducted by different organisations, where required sample sizes are smaller than the 45,000-requirement set out here. In self-select designs, there is no sample frame, instead the survey is advertised publicly, and respondents take part by choice rather than by invitation.

A voluntary self-select survey means that all unpaid carers in England would have the potential to form part of the sampling frame. This overcomes concerns related to other sampling frames as there is no risk of targeting households that do not have unpaid carers, which would incur additional costs and have possible low response because of increased ineligibility.

It would be possible to weight survey responses, if needed, using the assumption that all unpaid carers aged 18 years and over are eligible to answer the survey. Therefore, weights created could be considered non-response or final weights. The self-select method could be the most effective in cost and time because the survey could be advertised through targeted communications with carers' organisations and charities. This reduces address-based sampling costs associated with inviting people who are not eligible. The disadvantages of self-select sampling include the potential for respondents to be unrepresentative of the unpaid carer population.

Respondent-driven sampling

Respondent-driven sampling (RDS) is a "snowball" sampling method of surveying small or hard-to-reach populations. It is designed to provide statistically sound results without the need to conduct a large survey of the population to reach a small group. In the 1997 article, Respondent-Driven Sampling: A New Approach to the Study of Hidden Populations, published in Social Problems, Douglas Heckathorn introduced RDS with the primary aim of capturing rare or stigmatised groups. However its use has developed and is now used more widely to capture groups where there is no sample frame. 

The most important steps involved in a respondent-driven sampling approach include selecting a small convenience sample that are part of the population of interest. This initial sample is referred to as "seeds", they are then sent an invite asking them to pass the survey on to anyone they know that may be eligible. When a person is recruited, the seed and person recruited are often incentivised. The person recruited is then asked to recruit others to take part and this cycle continues until the required sample size is reached.

The most effective innovation is that through many waves of sampling, the dependence of the final sample on the initial convenience sample is reduced. This means that if there are enough waves of recruitment, this could be a more unbiased approach, particularly for underrepresented groups. However, this recruitment approach also assumes that those in the population of interest are connected by their networks, which can mean that individuals with poorer social networks have a lower probability of being reached.

RDS could be used to capture underrepresented groups of unpaid carers. This could be delivered after the initial sample design collection with the aim of contacting carers with certain characteristics to improve representation. This would involve monitoring management information throughout collection to determine which groups are underrepresented to target them.

While RDS can be used to capture underrepresented groups and account for bias, this has not been done in the context of unpaid carers before and would require testing to understand the effectiveness.

Methods for reducing bias

When using more novel approaches to sampling, such as those without a sampling frame, there are difficulties with measuring representativeness and weighting to account for any bias because of limited knowledge of the expected distribution. Data linkage is a possible solution, which could be used to assess representativeness as it means that data can be linked with other datasets that have good representation of the target population.

An additional option for highlighting and addressing potential bias is to run a small-scale random survey to assess the survey coverage and adjust for carers not selected. This could serve to collect data from these carers and to adjust for non-response bias of certain groups - either weighting by the proportions in the survey, or by estimating the total number of unpaid carers. 

A coverage survey can ensure that some unpaid carers not reached by a self-select or RDS design are accounted for. The results of a coverage survey can be used to calibrate the proportions of carers participating in the main survey, by characteristics collected.

A coverage survey can also be used to estimate the overall size of the unpaid carer population. The number of unpaid carers present in both designs can be used to estimate the number of total unpaid carers in the population in conjunction with weighting to population totals from the Census.

Administrative data

Besides surveys, unpaid carers are recorded in health and administrative data. Using administrative data to collect information about unpaid carers and the provision of services is a possible option for the future. Further research would be needed to explore what information could be analysed. However, current research, as published in the Health Foundation's Can you tell we care? article, indicates that the data available may not be sufficient at this point in time. Enabling an admin data approach would also take time to set up as it would likely require data-sharing agreements across service providers as well as updated contracts for services (to ensure clarity regarding the use of personal data). The time implications and availability of data on unpaid carers mean that a survey is currently a more appropriate method of capturing data. It could however be beneficial for the relevant organisations to work to improve administrative data over time so that it can be used as an effective source of data in the future.

The delivery method of a new survey should be considered as delivery can affect a range of factors, such as the quality and reliability of estimates from the data collected, as well as response rates. A review of the evidence and engagement with experts has enabled us to draw conclusions. Some high-level suggestions include:

• running the survey centrally rather than through local authorities would minimise resource and maintain consistency

• an online-first approach would be beneficial, as the paper option is resource intensive, expensive, and does not reach all carers (paper surveys on request will include those who cannot access the online survey)

• increasing the frequency of data collection as an annual survey would provide more frequent data to inform services (the Survey of Adult Carers in England is currently biennial)

• trying to optimise the time period of the survey for respondents in order to reduce survey fatigue and increase responses; this involves running the survey at a time in which other surveys are not in operation

It would also be important to consider a longitudinal design. This would enable the collection of data over time, with participants being involved over several waves. This type of design allows identification of changes over time, producing a richer assessment of the unpaid care sector. Longitudinal data would improve unpaid care intelligence and may enable causal inferences, while capturing further insight into how unpaid care and provision change over time.

Pilot study

Research and engagement with experts indicated a need for a survey of unpaid carers that captures a more representative view of the unpaid carer population compared with other surveys.

A representative sample frame of unpaid carers is not currently accessible, meaning that sampling designs other than targeted sampling should be tested to assess their effectiveness at capturing unpaid carers before implementation for a full-scale survey.

To do this, a pilot study to explore how a range of these methods work with the target population should be undertaken. This would involve using different sampling designs across different LAs and comparing results, with a focus on the data quality, representativeness and achieved sample size. If a data source were to become available as a feasible sample frame, targeted sampling should be tested alongside these methods. Potentially, the pilot study could highlight that blended sampling methods would be the most effective approach. Although this could require more complex analysis, it may improve insights into this group.

Evaluation of the sample designs would inform the design of a full-scale survey of unpaid carers. This would ensure any future survey designs are proven to work in the context of the unpaid care population.