2011 Census benefits evaluation report

Statistics producers are coming under increasing pressure to justify the cost of their work. They need to put together robust business cases setting out the benefits of what they do.

The logical and qualitative benefits of producing statistics are often clear. It’s easy to point out a number of essential uses for them, which are often underpinned by legislative requirements. However, statistics producers often need to go further than this and financially quantify the benefits delivered.

This is a challenging area because statistics in themselves don’t deliver benefits. It’s the use of statistics that delivers benefits through better, quicker decisions by governments, companies, charities and individuals.

It’s easy to describe these uses of census information, but the challenge comes in seeking to quantify the benefits in cash terms. However, governments need to decide spending priorities and statistics producers need to provide them with evidence to justify expenditure on statistics.

This paper describes how Office for National Statistics (ONS) valued the 2011 Census and demonstrated how the 2011 Census achieved an estimated annual benefit of £500 million.

Following the 2001 Census, the House of Commons Treasury Select Committee reviewed the census. One of its recommendations was that “ONS were unable to supply us with robust evidence to justify expenditure of over £250 million on the Census… We consider that any future Census should… be justified in cost-benefit terms”¹.

The National Audit Office says “evidence shows that two-thirds of public sector projects are completed late, over budget or do not deliver outcomes expected²” .

In light of these comments, we didn’t want to leave the realisation of the 2011 Census benefits to chance. We set out to ensure that benefits were realised and evidenced.

We started by carrying out a benefits mapping exercise. This links the “end benefits” to the “enablers” that had to be in place to deliver the end benefits.

In the case of the census, the end benefits arise when users make use of the published outputs to make decisions that help make peoples’ lives better. The enablers of this are:

• running a high-quality census
• users who are confident in the quality of the census data
• users who know what the data are and where to find them
• data are made available in easy-to-access ways
• support being available to users
• clear metadata that help users understand the outputs and use them appropriately
• outputs that are available in time for their main uses

We started the 2011 Census programme with the end benefits in mind. We planned backwards to ensure these benefits were delivered.

Notes for: Benefits management

1. For more information, please see The 2001 Census in England and Wales: Government response to the Committee’s first report of session 2001 to 2002.

2. For more information, please see National Audit Office. (2011). Initiating Successful Projects. Page 5.

To achieve quality statistics, the 2011 Census had to:

• maximise overall response rates
• minimise differences in response rates within and between local authorities

We therefore set quality targets, which included achieving an overall response rate of 94% and no local authority area or main population group with a response rate below 80%.

To do this, we radically changed our field design. We moved to a “post-out” system, delivering questionnaires to households by mail. We also targeted our field staff on areas predicted to have the lowest initial response (for example, inner cities).

We met the response rate target and significantly improved response in areas that are harder to estimate. This helped give census data users confidence in the quality of the 2011 Census data. For more information, see the 2011 Census General Report for England and Wales.

We started releasing data from the 2011 Census in July 2012. This was 15 months after census day and two months quicker than for the 2001 Census. By early 2015, we’d published more than 800 unique datasets with more than 8.2 billion cells of data. This compared to 360 datasets from the 2001 Census.

We made the census data more accessible to users and the wider general public than ever before. We did this by publishing the data online, which also helped to widen the census user base. We also presented data in more innovative ways, including infographics and data visualisations, and promoted the statistics using social media.

The census is highly valued by journalists and academics. The 2011 Census results featured extensively in the media. By hosting press media briefings (in July and December 2012 and January 2013) we were able to promote census releases to a wide audience. For the first release of data, there were more than 300 pieces of coverage in the national media.

This media coverage helped make users aware that 2011 Census data were available and also encouraged people to use the data. We had more than half a million page views of the census analyses and more than 600,000 page views of the census data visualisations. Even in 2016, the data were viewed around 1 million times.

We published more than 70 detailed articles containing census analysis on topics such as:

• migration and demography
• ethnicity and national identity
• health
• housing
• the labour market
• language
• religion
• unpaid care

These analytical articles focused on insights and trends. They generated media coverage throughout the publication process.

The 2011 Census continues to feed the public’s insatiable appetite for history and about who we are as a nation. According to BBC Wales, "the census is the gift that keeps on giving”, and journalist and academic Danny Dorling said, “the 2011 Census revealed a treasure-trove of facts we did not know about Britain”¹.

Notes for: Realising the census benefits – it’s all about getting users to use the data

1. For more information, please see the article Ending the national census would make us blind to our society (The Guardian, 2013).

To optimise the benefits from the census, we put significant effort into promoting its value to new or potential users. We also explained how they could benefit from 2011 Census data.

Local authority uses

We collected examples of how census data were being used by sending a survey to all 348 local authorities in England and Wales. Examples we received included the following:

• Lancashire County Council used 2011 Census data to target pilot programmes for its increased broadband access project: they also used the data for their Resilience Plan for the Heysham Nuclear Power Station area
• Bristol City Council provided an example of how they used census data to develop community learning; activities were targeted at learners who have few or no qualifications, and small area census data were essential in planning these community learning provisions
• a number of local authorities gave examples of how they used census data in public health campaigns
• most local authorities with social services responsibilities used census data for producing plans and strategies, needs assessments, and targeting and prioritisation
• Leeds City Council used census data to identify where there were more older people than average living in villages in an area; this provided evidence for spending £35,000 on projects that help older people live independently at home

Metropolitan Police use

The Metropolitan Police use census statistics to decide where to focus crime prevention efforts by mapping potential crime hotspots.

For example, in the London Borough of Bromley, age and housing statistics showed where pockets of people over 65 years old live. There were many cases where burglars were tricking elderly residents into letting them in by claiming they were from water or electric companies. Targeted crime prevention efforts reduced these incidents.

Promoting to representative and membership organisations

Representative and membership organisations were an important link with people from specific industries and professions. Census articles in membership magazines and on websites helped to reach professionals with a potential interest in census data.

For example, we recognised the information the 2011 Census provides on languages spoken would be useful for linguists. We placed an article in the Chartered Institute of Linguists membership magazine “The Linguist” and reached more than 7,000 members. Many of these people hadn’t used census statistics before.

Using case studies

We found using case studies was an effective way to illustrate how people and organisations can benefit from the 2011 Census. We used these case studies to provide examples of how different organisations from the private, public and voluntary sectors already benefit from census data.

We agreed the case studies with the organisations involved before publishing them on the 2011 census benefits pages of the ONS website. An example of this is how the London Fire Service uses 2011 Census data “to help find and help the areas of the community that are most at risk of fire”. Another example is how the Electoral Commission's work uses census data to help improve equality in practice.

Census data guidance

We made toolkits and factsheets available on the ONS website. These were designed to help get people started with census data.

The benefits of the census arise when the government, commercial sector, charities and others use census information to make better decisions than they could without it.

To evaluate the actual benefits achieved from the 2011 Census, we met with a wide range of users during 2012 to 2014 to:

• understand how census statistics were being used
• seek the information required to financially quantify the benefits
• calculate and agree benefit values

We gathered information from users in a variety of ways. These included:

• sending a questionnaire to the 348 local authorities in England and Wales
• holding meetings with policymakers in government departments
• meeting experts in particular fields
• speaking at user conferences
• running workshops
• direct requests for information

Techniques used

We used a number of techniques to financially quantify the benefits of the census. These varied depending on the use of the statistics, the data available and how easy it was to engage with users.

Whatever technique was used, a conservative approach was taken to census benefits valuation. Estimates were agreed with individual users, companies, or representative bodies.

Throughout the exercise of quantifying benefits we sought external assurance. Almost all the assumptions are drawn from external sources, not our own assumptions. We used an independent economist (a former Ministry of Defence Chief Economist) to validate our work.

Direct estimates from users

The simplest approach in terms of method was to get a direct estimate of the value of census data from users. It was easier to propose a value rather than ask them to estimate themselves.

Even so, they weren’t always able to react to the values we proposed. We found other approaches more successful.

Willingness to pay

A standard economic technique for estimating values of goods or services is to establish how much the recipient would be willing to pay for it.

We used this technique to value research based on census statistics. We contacted the major sponsors of policy research (usually government departments or other public bodies) and surveying universities (usually the major conductors of research) to find out what research they had carried out solely using census data and how much this research cost. We assumed that this cost was the value of the statistics. This is on the presumption that the output of the research is worth, to whoever commissions it, at least the amount spent on it.

Establishing willingness to pay is straightforward in theory, but we found it somewhat harder to apply for other uses for two main reasons. Firstly, although the question was posed to users as a genuine attempt to financially quantify benefit, some thought it meant we were considering charges for data. Commercial confidentiality was the second factor. Commercial organisations use census data to make business decisions and didn’t want to disclose how they use the data or how much value they attach to these uses.

Costs saved

An alternative approach we used was to establish what resources are saved because a census supplies the information users need. For example, if the census didn’t exist, would local authorities commission their own census or survey and how much would this cost? Would they commission research, or commit staff time to researching trends or updating previous censuses or surveys using modelled data?

One of the advantages of this technique is that estimating costs is usually easier than estimating benefits directly. Estimates can be worked out from known costs of staff time or purchased data. The financial benefit is the cost avoided by the census providing the statistics.

This will inevitably be an underestimate of the real value because it’s unlikely to represent the full benefit. It’s doubtful a council would spend, say, £100,000 on a survey if it only delivered £100,000 of benefit. They would probably want a return on investment of at least two to one to justify the expense (given funding constraints and other competing priorities).

Estimates will only relate to the value to the one user who commissions the work rather than the wider user base. Also, any surveys or work covering only part of the scope of a full census don’t bring the benefit of comparability that a national consistent exercise does. This therefore reduces the overall benefit.

We found estimating resources saved to be most appropriate to working out benefits for central or local government. This is because these users are often interested in a particular subset of the statistics. For central government departments, this may be particular variables. For local government, this may be the entire output set for the geographic area it has responsibility for.

In establishing the benefits of the 2011 Census, this approach was also used to estimate the benefit to the retailers in deciding where to open stores. Without population data, they would need to do their own expensive field work to find out about an area, rather than desk-based profiling based on statistics.

Contribution of census data (alongside other information)

In many cases, particularly those involving decision-making or investment appraisal, census statistics are used alongside other data and expert knowledge. The financial value of the decisions influenced by the statistics may be considerable, but this value can’t be attributed just to census statistics.

The question is: how much more efficient do census statistics make decisions? Or, what proportion of the expenditure is directly attributable to the census data?

We met with users to ask these questions. We found an effective way was to work out the proportions in workshops when the issues could be extensively discussed. We could then come up with a consensus amongst those present, and ask other data users to endorse or disagree with the findings.

Broadly speaking, there were two types of data use or decisions. These were:

• uses, decisions, or businesses founded on census statistics that couldn’t happen without the data (for example, businesses carrying out data interpretation and presentation such as specialist data analytics companies or consultancies)
• uses or decisions that are informed by census statistics to make them better but could happen without the census

For some sectors estimates were produced using both “top-down” and “bottom-up” methods. This gave a range and the mid-point value was chosen. For example, in the retail sector we had industry-wide estimates from experts (top down) and the cost of appraising one potential store site by one company (bottom up).

There are numerous benefits of census data that we weren’t able to financially quantify. Examples include:

• the benefit to individuals from application of the data, for example having a hospital or school within reach
• central government’s use of population figures as the denominator for numerous statistics expressed as rates, for example, the unemployment rate, mortality rate or teenage pregnancy rate
• addressing topical issues such as the long-term pension policy and unpaid caring
• the Office for Budget Responsibility (OBR) informing their forecasts, which guide the HM Treasury in determining the government’s fiscal policy
• local government data use in location planning of emergency services, including fire and ambulance stations, as well as route planning
• use by the NHS locally to plan and monitor service delivery
• use in equality impact assessments by central government, local authorities and health bodies to provide a base to compare the distribution of service users with

These are all essentially unquantifiable. There have also been other uses we have been unable to quantify due to commercial sensitivity. For example, mobile phone companies use data to plan services and profile customers. This market is very competitive and so firms are unwilling to share information about how they use census data.

Census data influences national debate. For example, the release of 2011 Census data sparked debates about the health of carers, changing religious affiliation, ethnic diversity, and migration, and informed general political debate. It isn’t possible to put a robust financial value on this debate, but the extent of media coverage and the discussion of these figures in election campaigns highlight their importance to society.

Public debate may also lead to changes in government policy and decision-making. The 2011 Census highlighted the poor health of carers. This issue was addressed by the Care Act 2014.

The assessment of the 2011 Census benefits concluded that the quantifiable benefits of the 2011 Census were an estimated £500 million per year. We’ve summarised the benefits by sector in the following table, with more detail in Annex A.

We believe we’ve made significant progress in the management of the 2011 Census benefits. This is both in helping to ensure that the benefits were realised, and in improving the quantification of benefits. This benefit information has helped underpin the business case for the 2021 Census.

Understanding the wide range of users and uses of census data is critical to being able to quantify benefits. If we aren’t aware of a data use, it can’t be valued.

Quantification of the benefits of official statistics such as the census is difficult. Users of census information find it hard to put a value on their use. This means national statistical institutes need to work closely with individual users and representative bodies (for example, trade associations) to understand the data uses and to explore alternative data that could be used.

Even then, financially quantifying the benefits of data uses is difficult, but some are particularly challenging. For example, good quality data influence scientific, national and informed general political debate. Public debate may also lead to changes in government policy and decision-making. However, it’s not possible to put a justifiable financial value on this debate.

Statistics are also highly valued by journalists and academics, but putting a financial value on this is impossible. Data produced independently and objectively are valued by users. However, this credibility is also hard to financially quantify.

The benefits to central government are particularly difficult to quantify – especially in policy development and evaluation. For example, how do you value the benefit to society from census information on ethnicity or religion, which helps the government put in place policies to support social cohesion?

This is clearly of great value, but it hasn’t yet been possible to robustly quantify it. In general, private sector uses of data are easier to quantify than public sector ones as success is measured financially. Therefore, we believe that our estimated benefits of census data to the public sector, and central government in particular, are still significantly underestimated.

We’re grateful to all users and experts who returned surveys, attended meetings and workshops or contributed to this work in other ways. Without their assistance it wouldn’t have been possible to carry out this work.

Details of the total benefit values we’ve been able to calculate are set out in Table 2. As noted previously in this article, it has been more challenging to quantify central government benefits than those from other sectors, so we believe these are significantly underestimated.

Details of the data uses included and the methods used to derive the values detailed in Table 2 are set out in Table 3.