Experiences of NHS healthcare services in England QMI

• Statistical designation: official statistics in development.
• Survey name: Health Insight Survey (HIS).
• Data collection: the HIS is an online survey of around 100,000 people aged 16 years and over in England.
• Time period: the HIS started in July 2024.
• Frequency: published every four weeks.
• How compiled: we derive estimates from questionnaire responses of HIS participants who were initially enrolled on the Office for National Statistics (ONS) Coronavirus (COVID-19) Infection Survey (CIS).
• Geographic coverage: England, with selected analysis broken down to the NHS regions of England, and integrated care boards.

This quality and methodology information report contains information on the quality characteristics of the data (including the European Statistical System’s five dimensions of quality), as well as the methods used to create it.

The information in this report will help you to:

• understand the strengths and limitations of Health Insight Survey (HIS) statistics
• reduce the risk of misusing data
• help you to decide suitable uses for the data
• understand the methods used to create the data

• The Health Insight Survey (HIS) is commissioned by NHS England and delivered by the Office for National Statistics (ONS).
• The HIS was launched in July 2024 in England to understand participants’ experience of their GP practice and other NHS services, including dental care and pharmacy services.
• Approximately 244,500 CIS participants were identified as being eligible for inclusion in the HIS and invited to participate.
• Achieved sample size in Wave 1: 104,109 participants.
• Only private residential households and their residents are included in the survey; people in hospitals, care homes and other communal settings are not included.
• We will be publishing weighted data every four weeks to summarise the responses received for each question in the survey.

Overview of the Health Insight Survey

The Health Insight Survey (HIS) was launched in July 2024, in England.

The HIS sample is drawn from participants in our Coronavirus (COVID-19) Infection Survey (CIS).

Participants were identified as eligible for the HIS if they:

• took part in the digital version of the CIS
• had agreed to be approached about other ethically approved research studies
• were aged 16 years and over on 1 May 2024
• had an address in England

More information on the CIS sampling method we used to select our participant pool is available in our CIS quality report: December 2022, which was last updated on 30 March 2023.

Participant recruitment

All eligible CIS participants were sent an invitation to the HIS by email. GOV.UK Notify, a service offered by the Cabinet Office, sent email communications on our behalf.

In many cases, more than one participant from a single household participated in the HIS. This is because the CIS was based initially on a random sample of households to provide a nationally representative survey.

Unlike in the CIS, no financial incentives were offered to participants for taking part. This could be a factor in the lower sign-up rates to the HIS, particularly among younger age groups.

During the survey, we might invite additional participants who were not part of the CIS to obtain a sample that is as representative as possible. Recruitment of new sample members will be carried out by initially contacting selected households by post. Those who wish to take part will be able to sign up online or by telephone. More information will be provided when this activity occurs.

Participant engagement

From the 244,500 individuals initially approached, 104,109 took part in the first wave of the HIS (a response rate of 43%).

Completion of the questionnaire was online only (paper copies were not posted to participants). However, if a participant required help with completing the survey online, a telephone number was provided so the dedicated contact centre could assist the participant and, if needed, complete the questionnaire on their behalf over the phone.

Uses of the HIS

The HIS provides important information about participants’ experiences of accessing care at their local GP practice, and their experience of other NHS services in England, including dental care and pharmacy services.

The statistics produced will provide valuable information to help the NHS to:

• improve local health services
• monitor their Delivery plan for recovering access to primary care, which aims to make it easier and quicker for patients to get the help they need from primary care

Users should be aware that data collected during some time periods may be affected by specific wider events, for example, the Microsoft Windows outage on 19 July 2024 and GP work to rule, effective from 1 August 2024. We do not expect this will affect the data quality, but it might cause unexpected trends in the outcomes reported.

Strengths and limitations

The main strengths of the HIS include:

• a large sample of participants, which allows for the production of estimates for smaller geographic areas
• a questionnaire which regularly examines various aspects of NHS primary health care and allows for changes in participant experiences to be monitored over time
• questions found to be unreliable in testing are improved and re-tested before being released into the field
• questions that have been cognitively tested to ensure they capture the intent of the questions, are easy to understand and reduce participant burden
• questions that have been developed in consultation with NHS England topic experts to meet user needs
• a quick turnaround, as datasets are published on our website four weeks after each wave has closed

The HIS sample, as with all surveys, is subject to possible bias. As the HIS study participants were taken from the larger CIS participant pool, the HIS sample is subject to the same biases as the CIS cohort. For example, participants in both surveys needed access to an email address to participate. For more information see ”Representativeness” in Section 5: Quality characteristics of the Health Insight Survey.

All estimates in our publications contain uncertainty. Although the statistics produced as outputs from the survey data are our best estimates, they will not be identical to the unknown true numbers we are trying to measure.

How we quality assure and validate the data

We quality assure our HIS survey data at each stage of the research process, including when we draw the sample, develop our questionnaire, and publish our data and estimates. These processes include:

• updating contact information in the sample daily through the HIS contact centre to reflect withdrawals or changes to personal details
• cognitively testing survey guidance and instructions to ensure the questions collect appropriate data
• iteratively testing questions until no further changes are made
• following a rigorous methodology for producing the weights used in the analysis
• testing the questionnaire, using test scenarios, to sense check questions and question routing
• conducting a series of checks on the data after collection to identify inconsistencies and invalid responses
• ensuring that a researcher reviews inconsistent or invalid responses before a decision is made about how to deal with them
• dual running of processes and checking estimates at multiple stages between analysis and publication (including the publication of final reference tables)
• applying statistical disclosure control and suppressing estimates to avoid any disclosure of personal information
• regularly monitoring and analysing feedback on the questionnaire to improve question wording and response options

This section describes the quality characteristics of the data and identifies issues that should be considered when using the statistics.

Relevance

The Health Insight Survey (HIS) seeks to understand how access to NHS services is changing over time, both across England as a whole and across different regions and demographics.

We collect data through self-completed online questionnaires. Data are analysed to understand how users access different NHS services and their perception of how these services are offered. This study analyses patients’ experiences of local NHS services, including GPs, dental care, and pharmacy services.

Once a time series is available and an evaluation of the statistics is complete they can be used to highlight potential pressures for the NHS and to help support wider services. For example, assisting governments with informed decisions on important policies, such as service planning, and understanding the impact of systems on different groups in the population.

Accuracy and reliability

HIS estimates in our publications contain some uncertainty. Potential sources of uncertainty include:

• the quality of the data collected in the questionnaire
• statistical uncertainty in the estimates, as the data are based on a sample of people rather than the whole population
• bias, which may not be fully addressed by weighting methods

To quantify uncertainty in our analyses, we present 95% confidence intervals in our data. Confidence intervals give an indication of the degree of uncertainty in an estimate. A wider interval indicates more uncertainty in the estimate. Overlapping intervals indicate that there may not be a true difference between two estimates.

We minimise errors during data collection, for example, missing data, by only enabling participants to submit the questionnaire after they have answered all the questions required.

Representativeness

Ensuring a representative sample of the general population is important for producing survey-based estimates broken down by characteristics such as age, sex, and NHS region. In the HIS, ensuring a representative sample helps us to understand trends in different population sub-groups across England.

Aspects of the study design may affect the representativeness of the study population compared with the wider population. For example, participants needed access to an email address to participate (this did not necessarily need to be their own and the survey could be completed by others on the participant’s behalf). This excludes a small proportion of the population from participating. The latest data from Ofcom's Online Nation 2023 report (PDF, 4.5MB) reports that 92% of households in the UK have internet access. We also know that 89% of households responded to Census 2021 online (for more information, see our Population and household estimates, England and Wales: Census 2021 bulletin, published in June 2022).

Unlike the CIS, no incentives were offered to take part in the HIS. This may affect response rates in some groups more than others.

In this section, we compare those that responded to the HIS with the target population. Note that the weighting described in Section 6: Methods used to produce the data aims to reduce the effect of these differences in the survey.

Within the HIS sample:

• females are slightly over-represented, while males are slightly under-represented (England 52% female and 48% male; HIS 57% female and 43% male)
• younger age groups (aged 16 to 24 years, 25 to 34 years and 35 to 44 years) are under-represented when compared with older age groups (aged 55 to 64 years, 65 to 74 years and 75 years and over), which are over-represented
• those reporting white ethnicity are largely over-represented (England 83%; HIS 96%)

Characteristics of the sample

Participants are asked for their employment status, and whether they work exclusively from home, another location or a combination of both. Further demographic information, such as age and ethnicity, is taken from the participants’ response to the CIS.

Accessibility and clarity

The Office for National Statistics’ (ONS’s) recommended format for accessible content is a combination of HTML web pages for narrative, charts, and graphs, with data provided in usable formats, such as Excel spreadsheets. Our outputs conform to the ONS Website accessibility policy in terms of formats, font sizes, and the presentation of tables and charts.

More details on our related releases can be found in Section 7: Related links. If there are any changes to our pre-announced release schedule, public attention will be drawn to the change and the reasons for it will be explained fully.

HIS data will be available in our Trusted Research Environment (TRE). As this provides access to microdata and more detailed data that have the potential to identify individuals, access to these data requires approved researcher accreditation.

Timeliness and punctuality

This publication provides timely and punctual information from the HIS, detailing our analysis on patients’ experiences of local NHS services, including GPs, dental care, and pharmacy services. These data are collected, processed, and published approximately four weeks after data collection ends.

Why you can trust our data

The ONS is the UK's largest independent producer of official statistics and its recognised national statistical institute. Our Data Policies and Information Charter details how data are collected, secured and used in the publication of statistics. We treat the data that we hold with respect, keeping the data secure and confidential. We use statistical methods that are professional, ethical, and transparent. More information about our data protection policies is available.

Provisional estimates and revisions

When any HIS data are found to be in error, both the data and any associated analysis that has been published by the ONS will be updated in line with our Revisions and corrections policy.

There are several reasons why we may wish to update the survey estimates or datasets once they have been published; for example, if errors are discovered in raw or derived variables.

Every effort is made to check the data thoroughly before they are published. However, errors do occasionally occur. When errors occur, corrections are made in a timely manner, announced, and clearly explained to users in line with the ONS guide to statistical revisions. Work is also undertaken to prevent the same error from occurring again, for example, by reviewing and improving code.

The data collected by the Health Insight Survey (HIS) enable us to estimate how people access NHS GP services, dentists, and pharmacies as well as their perception of these services.

How we collect the data

Sampling method

Our overview of the HIS in Section 3: Important points outlines the participation criteria for the survey. Coronavirus (COVID-19) Infection Survey (CIS) participants were considered eligible to be invited to HIS if they:

• took part in the digital version of the CIS
• had agreed to be approached about other ethically approved research studies
• were aged 16 years and over on 1 May 2024
• had an address in England

More information on the initial CIS sample from which we selected our participant pool is available in the CIS quality report: December 2022 which was last updated on 30 March 2023.

Study design

Participants have been divided into nine equal groups. Each household has been assigned randomly to a group, and each group had a similar distribution of main characteristics (for example, age).

Each week, for three consecutive weeks, participants in three groups are sent an email invitation (on Tuesday, Wednesday, and Thursday) to complete the online questionnaire. Participants have a seven-day window, including the day their email was sent, to complete the questionnaire. In the fourth week, no new email invitations are sent out. This provides a short period between participants completing the questionnaire and new invitations being sent, for updates to our systems and processes to be implemented. This study design means that each participant is emailed approximately every four weeks with an invitation to participate. The study follows a panel design which means that the full sample is emailed every four weeks.

Each four-week period is referred to as a “wave”. The seven-day questionnaire windows overlap, as this helps to smooth out the number of responses received each day. Participants were more likely to return the questionnaire at the beginning of their window.

Participants who do not complete the HIS questionnaire three waves in a row will be removed from the survey sample.

Data we collect

We collect data from each participant by using an online questionnaire. Participants are asked a series of questions on their ability to access NHS GP services, dentistry services and pharmaceutical care. They are also asked questions about their subjective perception of these services.

Response rates

The response rate for Wave 1 of the HIS was 43%, a further breakdown of the response rates by participant characteristics are included in Table 40 of the data tables accompanying our Experiences of NHS healthcare services in England: September 2024 bulletin. Response rates for subsequent waves will be added to this table in each publication.

How we analyse the data

Allocating NHS geography

A participant’s registered GP surgery is identified using the Patient Demographic Service. A lookup file is then used to attach the primary care network (PCN) to which the GP surgery belongs. Higher level health geographies are then mapped on through the PCN. This is used to ensure participants nest correctly in the hierarchy (for example, the number of participants in each integrated care board (ICB) is the same as the sum of all participants in the PCNs that are located in each ICB).

In some cases, a participant may be allocated to an ICB that is different to the one that covers the area where they are recorded as living. We carried out analysis to determine whether this makes a difference to the number of participants in each ICB. We found that most participants were allocated to the same ICB if they had been mapped using their home postcode.

There are a small number of participants that we are unable to link to a GP surgery (under 1%). Questionnaire responses from these participants are included in the overall figures and the demographic breakdowns but not the geographic breakdowns.

Weighting

We apply weighting to our HIS survey results to ensure they are representative of the target population. This adjusts for differential inclusion or response rates of demographic groups, which could otherwise result in under-representation (for example, of younger age groups).

The weighting strategy accounts for:

• the probability of inclusion in the CIS
• the probability of selection for the HIS
• the probability of responding to the HIS
• known population totals

The HIS sample is a subset of the CIS sample. Therefore, the CIS design weights, which account for an individual’s probability of inclusion in the CIS, are used as a basis for the HIS design weights. An individual’s CIS design weight is then adjusted for factors correlated to the likelihood of the individual consenting to further research, derived using logistic regression. Logistic regression is also used to adjust for the probability of sampled participants responding to the survey.

Initial weights are then calibrated to population totals. Calibration groups are age group by sex and age group by health region separately.

Probability of being selected to take part in the HIS

Design weights are used to account for the probability that a person has been selected to take part in the survey. Differences in the probability that a person is selected to take part in the survey can introduce bias to estimates if they are not accounted for.

The HIS sample is a subset of the CIS population. Therefore, the probability of being selected for the HIS is linked to the initial probability of being selected for the CIS. For this reason, the CIS design weights are used as a basis for the HIS design weights.

Participants need to fulfil certain requirements to be selected from the CIS sample for inclusion in the HIS sample (for example, having access to the internet to complete the questionnaire). This means that their probability of selection is one. However, there are factors that influence the likelihood that participants will fit those selection criteria; for example, some participants are more likely to respond online or consent to further research. To account for this, logistic regression is used to identify the probability of a participant being selected for the HIS, given they took part in the CIS. The regression model includes:

• age
• sex (male and female)
• region
• ethnicity (as two categories: “White” and a combined category comprising all other ethnic groups)
• Index of Multiple Deprivation quintiles (IMD)

For each participant, the CIS design weight is multiplied by the reciprocal of the probability of selection for the HIS, given they took part in the CIS, to produce the HIS design weights.

Probability of responding to the HIS

There is non-response within the sample that were invited to the HIS. We account for this using a logistic regression model. We use information from previous survey responses that is not available as population totals, for example, previous ethnicity. Therefore, a logistic regression model is used to calculate the probability of response. The model includes:

• age
• sex (male and female)
• region
• ethnicity (as two categories: “White” and a combined category comprising all other ethnic groups)

HIS design weights are multiplied by the reciprocal of the probability for responding to create an initial weight. These weights are the basis used for calibration.

Calibration and population totals

Calibration is carried out to ensure that the weighted samples for the different population groups sum to the known population totals. Calibration can account for more than one set of population totals, summing to the same number, at the same time (for example, population totals based on age by sex, and population totals based on health region for the same population). This avoids weights being based on small sample sizes, for example, when looking at age by sex by health region.

Here, the calibration groups are age group by sex and health region separately.

To obtain population totals for the period closest to the study dates, ONS population projections were used rather than the GP registered population. The population protection totals will differ to the GP registered population, particularly in areas of the country with higher levels of population movement. The ONS population projections were selected as they have been adjusted to remove those in communal establishments and are more suitable for the questions unrelated to GP access (for example, pharmacy and dentistry questions). The calibration population totals are provided in Table 41 of the data tables accompanying our Experiences of NHS healthcare services in England: September 2024 bulletin.

Figure 1: Health Insight Survey sample selection and weighting flow diagram

Source: Health Insight Survey from the Office for National Statistics

Download this image Figure 1: Health Insight Survey sample selection and weighting flow diagram

.png (100.0 kB)

More information about the Health Insight Survey and why it was created is available on our About the Health Insight Survey web page.