1. Summary

This article presents new proposed methods for quality adjusting the Office for National Statistics’s (ONS’s) measure of output of public order and safety (POS), a component of public service productivity. It includes looking at a single quality adjustment measure applied to the conviction and correctional services of the criminal justice system as a whole, based on a severity adjusted measure of total re-offences per offender. As well as this, the article looks at specific quality adjustments to the various components of these services, to reflect changes in specific outcomes. This article represents the first step in developing these methods and is subject to change, following further development and consultation, prior to implementation.

Nôl i'r tabl cynnwys

2. Introduction

The aim of this article is to seek user feedback on the measurement and application of a quality adjustment to the public order and safety (POS) component of public service output, which represented 4% of total UK public service expenditure in 2014.

POS has four main components: fire-protection services, courts (which itself has five further sub-components), probation, and prisons1. All of these are currently measured on a quantity output basis. This involves measuring the quantity of different activities performed and functions provided by each service, such as the number of prisoners held in custody. These are then cost-weighted2 together to produce an aggregate quantity output index for POS. However, such measures may not strictly capture changes in the quality of those activities, as set out in the Atkinson Review.

The Atkinson Review, which, having developed a framework for measuring the output and productivity of public services, argued:

“the output of the government sector should in principle be measured in a way that is adjusted for quality, taking account of the attributable incremental contribution of the service to the outcome.”

Despite this, in the current annual public service productivity articles, quality adjustments are applied to around half of total public service output, accounting for changes in the quality of healthcare and education services.

This article will focus on extending coverage of the quality adjustment to include the output of the criminal justice system element of POS. This includes the criminal court system3 (magistrates’ courts, Crown Courts, and Crown Prosecution Service and Legal Aid), prisons and probation, as set out in Table 1.

A quality adjustment will not be applied to fire-protection services and county courts. This is as these services are deemed to have different outcomes from the rest of POS and so cannot be captured by the quality adjustment methods discussed in this article.

The first quality adjustment method presented in this article is uniformly applied to the selected components of POS listed in Table 1. This method treats criminal justice as one interlinked system, of which the primary target outcome is to reduce re-offending by providing appropriate disposal and rehabilitation services. The second quality adjustment attempts to account for the fact that these subcomponents may have specific target outcomes, in addition to reducing re-offending, such as ensuring prisons remain safe.

The time series covered by these quality adjustments is from 1997 to 2014, in line with the latest annual estimates of public service productivity, with the data used to create them introduced incrementally over time as suitable data becomes available.

Notes for: Introduction

  1. The police service is not included as part of the POS category in this publication and the annual public service productivity release. This is because policing has multiple outcomes relating to POS as well as solving crime and participating in the criminal justice system. We propose to consider this area in a further article treating policing as a separate service area, which will not be covered in this publication.

  2. Cost-weighting uses the expenditure on different activities as a proxy for the value of those activities to weight different activities together, using their individual unit cost or share in total expenditure. Therefore the more high-value activities, as measured by those that more money is spent on, should contribute more towards the overall growth in output.

  3. Ideally, we would separate out the civil elements of magistrates’ courts, however, it is not simple to do so using current measures and so a quality adjustment will be applied to all of the magistrate’s courts system.

Nôl i'r tabl cynnwys

3. Re-offending quality adjustment

The first method of quality adjustment uses severity adjusted total reoffences per offender as an overall measure of quality of the criminal justice system. An important social outcome of the system is to reduce the volume and severity of further crimes committed by those who have gone through the criminal justice system, therefore this adjustment attempts to measure the effect the system has in achieving this. A criminal who is convicted and then does not go on to re-offend, who otherwise would have, demonstrates the system is successfully achieving its intended outcome of reducing crime. This scenario would therefore reflect an improvement in quality of services, captured by a re-offending quality adjustment. To account for this, a quality adjustment measure is calculated using the method set out in Equation 1.

Equation 1:

Data on re-offending covering 2000 to 2014 is taken from the Ministry of Justice proven re-offending statistics for England and Wales, with re-offending capturing any offence committed in a one-year follow-up period, by both adult and juvenile offenders1,2. We use this data to construct a measure of the total number of re-offences per offender in the cohort for each year. Even though the outcome we are trying to capture is the impact of re-offending on society and how this changes over time, the quality adjustment measure we use is on a per offender basis. This is to reflect the quality of each unit of quantity output of the criminal justice system. The effective quantity output in this case is individual offender disposals, which is already captured by existing measures of output for each component of the criminal justice system used in public service productivity.

As can be seen in Figure 1, the measure of total re-offences per offender (excluding adjustment for severity) shows a general downwards trend from 2002 to 2009, driven by falls in both the portion of offenders who re-offend and average number of crimes committed by those who do re-offend. Following this, the index does show some growth but remains below its peak in 2002.

To further account for changes in the impact of re-offending on society, an adjustment is made to reflect changes in the severity of these re-offences. Severity of the crime committed is an important component of the social experience of crime; therefore, a higher share of more severe re-offences reflects a worsening of outcomes for the criminal justice system. Applied by weighting together categories of re-offences, the adjustment for severity uses weights derived from the Office for National Statistics (ONS) Crime Severity Score for England and Wales. More information on this source, as well as implied category weights, can be found in Annex 1.

Applying this to our initial metric, Figure 1 shows that adjusting for crime severity has little observable impact in the initial period, with both measures tracking closely to each other on a general downwards trend. However, from 2007 onwards, we observe a divergence in the two series, with the severity adjusted index growing at a consistently higher rate. As a result, by 2014, there is a difference of 5.2 percentage points. This demonstrates an increase in the share of more serious crimes committed by re-offenders. For example, summary crimes (which are the least severe group) represented 45% of crimes committed by re-offenders in 2007, but by 2014 this decreased to 37%.

From this metric, we can then observe how the relative level of re-offending, accounting for the severity of the offence, has changed over time. To use this to reflect changes in quality the inverse of it is taken. This implies that a decline in re-offences or reduction in the severity of offence would reflect an improvement in quality. Figure 2 presents this quality index and shows that from 2000 to 20142 the quality of the output of the criminal justice system increased. Within this period, there was a significant improvement in quality of 33% from 2002 to 2009. Following this, however, quality began to fall, driven by an increase in the rate of re-offending and a worsening in the severity of crimes committed.

Notes for: Re-offending quality adjustment

  1. Re-offending captures "any offence committed in a one year follow up period that resulted in a court conviction, or caution in the one year follow-up or a further six month waiting period.” This follow-up period begins from when the offender is released from custody, receives a non-custodial conviction at court or receives a caution, with the measure covering a cohort of individuals who begin this follow-up period in a given 12 months.

  2. Data for 2001 is not available and so are imputed using an average of 2000 and 2002 data.

Nôl i'r tabl cynnwys

4. Limitations of re-offending quality adjustment

The quality adjustment of government output is a particularly challenging area, given the intrinsic difficulty, their subjective nature and the relatively limited experience – both in the UK and across the world. It is to be expected that the initial measures will be approximate and contain only partial information.

One potential issue with using data on re-offending is that this could change as a result of factors unrelated to the quality of the criminal justice system. This could be as broad as the changing characteristics of individuals within the cohort for which re-offending is measured. If these characteristics make an individual inherently less likely to re-offend, an increase in the share of these individuals over time would lead to a fall in the re-offending rate – assuming all else remains constant. This then would not reflect an improvement in quality of the criminal justice system services but reflect changes to the makeup of the cohort.

To control for these offender characteristics the Ministry of Justice historically used a model that attempted to produce a re-offending rate that controlled these changes. An adjusted re-offending rate calculated using this model is presented in Figure 31 and this is compared with the actual (unadjusted) re-offending rate. The re-offending rate differs from the severity adjusted total re-offences per offender measure we use, as it does not account for changes in the number of crimes committed by those who do re-offend or the severity of those crimes. However, it can still be used as a way to assess the effect of changes in the cohort on re-offending.

Comparing these series shows a sizeable discrepancy between the actual rate of re-offending and the rate controlling for offender characteristics. The re-offending rate adjusted for changes in the cohort shows an increase in the re-offending rate of 1.4 percentage points from 2002 to 2012. In contrast, the unadjusted re-offending rate declines by 2.9 percentage points over the same period.

This suggests that the decrease in re-offending from 2002 to 2009 – as shown in Figure 1 – may be partly as a consequence of changes in the cohort rather than improvements in the quality of the output of the criminal justice system.

It is also worth noting the limitations in the proposed method to adjust for the changing severity of crimes. As well as being unable to control for changes in cohort characteristics, similarly to that referenced earlier, the Crime Severity Score for England and Wales currently only allows us to create severity weightings for eight groups of crimes, with weightings determined by crimes committed by the population as a whole in the 12 months to March 2016. As these weightings are not specific to re-offenders and at only one point in time, re-offenders in the cohort could have a different distribution of crimes committed for that year and could have a different distribution within these groupings over time. However, these factors would not be captured by our severity adjustment measure.

Notes for: Limitations of re-offending quality adjustment

  1. Using data from Proven re-offending statistics: January 2012 to December 2012.
Nôl i'r tabl cynnwys

5. Specific quality adjustments

This section introduces a range of additional measures of quality in the criminal justice system, reflecting the unique elements of quality of the various subcomponents of the criminal justice system. These are all used in combination with the re-offending quality measure, with an overview of these quality adjustments measures set out in Table 2.

Nôl i'r tabl cynnwys

6. Prison safety

The first of these specific quality adjustments looks at the safety of prisons. This aims to reflect that safety of prisons is an important component of the quality of the output of the prison service, as set out in the Prison Safety and Reform White Paper. To therefore account for this, measures taken from the Ministry of Justice Safety in custody statistics are weighted together. These measures are: assaults on prisoners and staff; homicides; self-harm incidents; and self-inflicted deaths.

The first step in this methodology is to group incidents by relative severity. The first group includes serious assaults and self-harm incidents requiring hospital treatment1, which are summed together to create an aggregate series of severe incidents in prisons. The remaining assaults and self-harm incidents are summed to create the second group, referred to as less severe incidents, while data on self-inflicted deaths and homicides forms the third group. All three series are then converted to a per prisoner population basis, as we are interested in the quality of each unit of output – which in this case is the number of individuals held in custody.

Figure 4 presents an index for each of the three series. It shows that despite some volatility there has been a general downwards trend in the number of self-inflicted deaths and homicides per 1,000 prisoners, followed by a period of relative stability until 2012. Subsequent to this, however, there has been consistent year-on-year increases in self-inflicted deaths and homicides, rising to 1.43 per 1,000 prisoners in 2016 – a level over double that in 2012 and above the previous 1999 peak.

Data on self-harm incidents and assaults show that until 2012 the number of incidents remains relatively flat in both the severe and less severe categories. However, from 2012 onwards, there are substantial increases in the number of severe incidents, having reached over double the 2012 level by 2016. The increase in less severe incidents is less substantial; however, it is still over 70% higher than the 2012 level by 2016.

Converting these individual series into a single index requires weighting together dependent on their relative severity. This is achieved by using the total cost to society of workplace injuries in 2014 to 2015, taken from the Health and Safety Executive, as weights. The more severe incidents are given a weight based on non-fatal injuries, for which there are seven or more days’ absence, while less severe incidents are weighted by when there is six or fewer days' absence absence. Self-inflicted deaths and homicides use weights for fatal injuries. While only reflecting the relative values of the total cost for different types of injury, this still assumes that the relative severity of workplace injuries is broadly reflective of incidents in prisons.

To then create an index of quality change, the inverse of the combined index of injuries and deaths is taken. Again, this is to reflect that the relationship between our measures used and quality is inverted, as an increase in the number of injuries and deaths in prisons would imply a decrease in associated quality. Figure 5 illustrates this quality index. This shows that safety declines initially, followed by a fairly flat period to 2004. These changes in the quality measure up to 2004 are driven entirely by changes in self-inflicted deaths and homicides, as consistently reported data on assaults and homicides are not available before this and so the number is assumed to be constant for these years.

Data on the number of self-harm incidents and assaults is incorporated into the quality series in 2004 and 2005 respectively. From 2005 to 2009, there is some volatility in the quality series, with a net improvement in level of quality, although this is still driven largely by changes in self-inflicted deaths and homicides rather than the inclusion of injuries. The index then remains fairly flat until 2012, after which quality falls substantially, driven by increases in all three components.

Notes for: Prison safety

  1. Incidents requiring hospital treatment is used as an estimate of severe self-harm incidents. However, whether a prisoner is taken to hospital as a result of a given incident depends on factors in addition to the incidents severity, such as local facilities available at the establishment.
Nôl i'r tabl cynnwys

7. Escapes from prisons

Ensuring prisons fulfil the role of public protection is also an important objective of the criminal justice system. The measure used to assess how the prison service is performing in achieving this objective is the total number of escapes from prisons, National Offender Management Service prisoner escorts and contractor escorts1. These figures are published in the supplementary tables of the Prison performance statistics release.

Figure 6 shows how the number of escapes per 1,000 prisoners has changed over time2. This is on a per 1,000 prisoner basis as we are interested in the quality of each unit of the quantity output of the prison service. This measure shows that between 1997 and 2010, the number of escapes fell, although the rate of this decrease slowed over time. After 2010, the number of escapes per 1,000 prisoners remains relatively flat.

Once again, we create an inverse of this index to reflect how quality is changing, so that increases in escapes reflect decreases in quality. This inverse index is presented in Figure 7 and shows a steady increase in the quality measure from 1997 to 2004, followed by a faster increase in quality until 2010. Following this, there is some volatility in the inverse measure, resulting in a net decrease by 2016.

This trend growth rate for Figure 7 shows a notable difference to the trend growth rate of raw number of escapes per 1,000 prisoners, which shows the greatest decline in escapes to be from 1997 to 2005. Contrary to what might be expected, the quality measure is only growing slowly over this period. This is as the scaling that occurs when an inverse index is created means that relative changes in the number of escapes are captured, rather than the absolute change in the number of escapes.

As a result, the quality index in Figure 7 arguably underestimates the improvement in quality before 2004, which was a large absolute decrease in the number of escapes but a small relative change. In contrast, from 2010 onwards, the absolute number of escapes is very small. This means that small changes in this absolute number are big relative changes and as a consequence can lead to sizeable changes in the quality index, despite only being small changes in the raw number of escapes.

To try to overcome these issues an alternative quality index is created. This uses the Ministry of Justice historic target for an escape rate below 0.05% of the prison population as a baseline. The actual number of escapes is then subtracted from this baseline and the remaining number converted to an index and used as a measure of quality. Figure 8 shows this alternative quality measure. This has a steeper initial increase in quality than the index using a basic inverse, with the year-on-year increases in quality slowing until 2010, after which the index remains relatively flat.

As this measure better reflects the changes in the raw numbers of escapes per 1,000 prisoners, this alternative quality measure will be used in Sections 8 and 10 of this article to assess how the quality of this aspect of prisons is changing over time.

Notes for: Escapes from prisons

  1. Absconds from open prisons are not included as this only applies to prisoners held in open prisons which are a subset of prisoners

  2. The number of escapes is converted from a financial year basis to a calendar year basis using a cubic spline.

Nôl i'r tabl cynnwys

8. Overall prisons quality measure

These quality metrics can then be brought together, along with the quality measures using severity adjusted re-offending, to reflect the combined change in quality associated with prisons. This is done by taking an average of the three series, weighted using the relative measures taken for public protection (used to weight escapes), reducing re-offending and decency (used to weight safety) from the 2015 to 2016 prison reporting system.

Figure 9 presents the weighted combination of the three quality series. This shows an initial decrease in overall prison quality from 1997 to 1999, due to increases in self-inflicted deaths over this period. From 1999 until 2009, quality follows a general upwards trend, as all three quality measures increase over this period. From 2012 onwards, there is a decrease in quality, driven by a fall in the safety quality measure.

Nôl i'r tabl cynnwys

9. Courts’ timeliness

Isolating criminal courts as a sub-set of the system, an aspect of quality specific to them is their timeliness in arranging and completing trials.The delivery of sentences in a timely manner is favourable in many respects, for example, allowing those who are found not guilty to return to the community in a timely manner. However, this must also be balanced with ensuring fair treatment of suspects and victims and allowing the appropriate time for preparation of criminal cases.

Looking to reflect changes in the average time from start to completion, the data used to construct a measure is from the Ministry of Justice Criminal Court Statistics, which provides “timeliness” measures for magistrates’ and Crown Courts. For magistrates’ courts, we use the mean time taken from Charge or Laying of first information to completion. For Crown Courts, a combination of average waiting time1 and the mean time from the start of the main hearing to its completion are used. Annual data is only available from 2011 onwards; therefore, prior to this, we assume that the “timeliness” of court cases is constant.

Comparing the average number of days taken by each respective court, Figure 10 shows that Crown Courts’ proceedings take consistently longer to reach completion relative to magistrates’ courts. This would be expected due to the different nature of the cases dealt with and does not represent a difference in quality. However, from 2013 to 2015, there is an increase in the time taken for Crown Courts’ cases to reach completion. This in theory would suggest a worsening in quality of Crown Courts over this timeframe, although it is important to note this increase in time taken coincides with committal proceedings being abolished in 2013. This caused an increase in Crown Courts' receipts, which would have increased waiting times while additional cases were being cleared. In contrast, the time to completion is relatively unchanged for magistrates’ courts over the period, suggesting a fairly constant quality of service.

This data can then be used to construct two quality measures, with increasing times to completion represent decreasing levels of quality and it is therefore once again necessary to create an inverse of these measures to reflect quality. This quality index is presented in Figure 11 and shows that while the two series track each other fairly closely at the start, there is a substantial drop in quality in 2014 and 2015 for Crown Courts, although this coincides with committal proceedings being abolished.

Similar again to other services, these “timeliness” quality measures for magistrates’ and Crown Courts can then be combined with data on re-offending to produce an overall quality index for courts. This is done using equal (50/50) weighting.

Notes for: Courts’ timeliness

  1. The data used to estimate waiting times is constructed from a different source to the other data on timeliness and is used to provide a more complete coverage of defendents.
Nôl i'r tabl cynnwys

10. Results

To observe how the proposed quality measures would adjust output of public order and safety (POS), we can apply each measure to the appropriate component of POS before aggregating together.

The severity adjusted re-offending measure is applied to the components of POS identified as part of the criminal justice system, as set out in Table 1. The combined prisons’ quality adjustment is applied output of the prison services, while timeliness of courts is applied to the output of their associated court type. This means that some areas have two quality adjustments applied. In these incidences, the quality adjustments are applied on a weighted average basis, with Table 3 outlining the weights used. To then aggregate together, all the components of POS, including non-quality adjusted components, are cost-weighted together to produce an aggregate index for POS quality adjusted output.

Bringing all of these together, Figure 12 compares the existing quantity output of POS with the same output series adjusted for changes in quality as highlighted in this article.

Over the period, we can see that, despite some initial growth, the quantity output for POS services has been on a general downwards trend since 2003. As a result, quantity output is around 12.9% lower than in 1997. However, when taking into account changes in quality, we do see a somewhat different story. Rather than contracting in the early 2000s, quality adjusted output continued to grow, reaching a new peak in 2006. However, later in the period, this trend begins to change, with quality adjusted output falling from 2009 onwards.

Figure 13 sheds further light on the drives of growth in quality adjusted POS output, breaking down into contributions from its respective components. Once again, we can see that for most of the last 11 years, quantity output in POS has been contracting. However, this was (for the most part) more than compensated for by improvements in the quality of the output – in particular when taking into account changes in severity adjusted re-offending. However, from 2010 onwards, we can see that, as well as quantity output falling, improvements in the quality of output begin to lessen or, in fact, worsen, having an overall negative impact over the last five years observed.

Nôl i'r tabl cynnwys

11. Quality measures not used

Other measures were considered to be included alongside the measures discussed in this article but were deemed unsuitable as measures of quality.

For prisons, this includes data on accredited courses completed by prisoners. This measure was not consistent over time and in theory the effectiveness of these courses should be partly captured by re-offending data. Overcrowding of prisons was also assessed as a measure of quality. However, this was also not used, as the optimal levels of prison capacity were deemed to be dependent on prison design rather than measures of capacity available. Safety in custody statistics were therefore decided to be a more reflective measure of safety in prisons. In addition, we looked at using absconds from open prisons, however, chose not to include this as it only applies to a limited subset of prisoners.

Additional quality measures for the other subcomponents of the criminal justice system were also considered. We looked at serious further offences (SFO), notifications that resulted in a conviction as a measure of quality specifically for the probation service. However, the process for triggering a serious further offence notification changed multiple times from 2006 to 2009. In addition, since then the basis for carrying out some types of SFO review means this is not a consistent measure over time of the number of serious offences committed by offenders on probation.

We also considered additional quality measures specific for courts. This included using type 1 errors, when an innocent individual is wrongly convicted, as a measure of quality. This could, in theory, be assessed using data on the proportion of convictions that are successfully appealed. However, there is the issue of how reflective this is of type 1 errors, as the number of appeals could be driven by factors such as sentencing severity (that is, the sentence of guilty was correct, but the sentence length or nature was successfully challenged). In addition, capturing type 1 errors without capturing type 2 errors (where a guilty individual is not convicted) means quality could be rising due to higher thresholds for conviction. This would have the negative effect of higher levels of type 2 errors, which could be deemed to be a worsening of quality.

Nôl i'r tabl cynnwys

12. Next steps

This article presents a set of proposed methods by which to capture changes in the quality of services of the criminal justice system – on both a holistic and sub-component basis – and approaches by which to implement them as part of the Office for National Statistics’s (ONS’s) measures of UK public service productivity. These estimates represent the first step in developing this new method and provide an idea of how quality for the service may have changed over time.

We propose to publish estimates using this method early next year, expanding the coverage of quality adjustments in UK public service outputs. Estimates will be published as part of the annual public service productivity article – Public service productivity estimates: total public service, UK: 2015 – which brings together our measures of inputs and outputs for different public services and latest estimates of total UK public service productivity.

We acknowledge this is a first step and are keen to develop such measures further. We are therefore keen to:

  • discuss the methods and results presented in this article, particularly in relation to the suitability and completeness of the underlying data used
  • receive feedback on our proposed methods in both aggregation and implementation of the quality indices.

Feedback on the use of these estimates and suggestions for improvements will be essential for the future development. All feedback is welcome and can be sent to productivity@ons.gov.uk.

Nôl i'r tabl cynnwys

13. Annex: Implied severity weights

The implied severity weights used are shown in Table 4. These are from the Office for National Statistics (ONS) crime severity score.

To do this, we take the total of the offence rate and the crime severity score for the year ending March 2016 and apportion both out based on the shares in Annex 1 of the Crime Severity Score publication. We then divide the apportioned crime severity score by the number of offences to get the severity of the average offence in each grouping. One limitation is that summary crimes are not separately captured by the Crime Severity Score and so are given an arbitrary weight half that of the least serious offence group.

These weights can then be applied to the breakdown of re-offences published in the proven re-offending statistics, as they both use ONS crime classifications. The “Other crimes against society” category is applied to the sum of the following classifications by which re-offences are categorised: drug, possession of weapons, public order, miscellaneous crimes against society, and “other”.

Nôl i'r tabl cynnwys

Manylion cyswllt ar gyfer y Erthygl

Sam Turnock
productivity@ons.gov.uk
Ffôn: +44 (0) 1633 456305