Data Standards Policy

This Data Standards Policy describes the responsibilities and requirements for the development, maintenance and implementation of data standards within the Office for National Statistics (ONS).

The policy applies to all ONS employees, including staff on fixed-term, temporary or permanent contract, staff on secondment, students and contractors.

Data standards are a set of well-defined rules by which data are described, recorded and shared in order to ensure common understanding among data users and to maintain data quality (integrity, consistency, format and meaning). They can take different forms depending on what they describe: assertions about how a field must be populated, rules governing the relationships between fields, detailed documentation of acceptable and unacceptable values, format, and so on.

Data standards provide the rules by which data can be cleaned; enforcing these standards promotes consistent results from the processes using them. The use of common terminology and consistent data formats promotes re-usability and facilitates data sharing, both within the ONS and across government. It facilitates more effective and efficient matching and linkage and improves data quality.

This Data Policy supports the Data Principles which underpin the Data Strategy.

Data standards are documented agreements on the format and definition of data, including guidance on their use. Where possible, existing industry data standards shall be adopted before new ones are developed. It is the responsibility of the Data Standards Data Architect to keep abreast of changes to external standards which have been adopted, ensuring that they are still suitable for the ONS, and updating guidance on their use if necessary. The ONS shall develop its own data standards only when suitable external standards do not exist.

Before being made available for use, data standards are reviewed by the Data Architecture Review Panel and are formally approved by the Lead Data Architect. Approved data standards are listed in the Data Standards Catalogue, alongside a backlog of data standards waiting to be developed.

Compliance with data standards is carried out by the teams processing the data, using a self-assessment process as agreed by the data standards data architect. The data standards data architect shall carry out spot checks on the recorded self-assessments. If a team has problems conforming to a data standard, a concession can be requested. If granted, concessions shall usually be time-bound and recorded in a centralised list.

The Data Standards Authority provides standards and guidance, which facilitate data sharing across government. These complement the ONS data standards and must also be adhered to.

Development and maintenance of data standards

Data Architects shall develop data standards based on specific business needs. Standards shall be developed in consultation with stakeholders and subject matter experts (SMEs). Where possible, existing industry data standards shall be adopted before new ones are developed; the ONS shall develop its own data standards only when suitable external standards do not exist.

Where industry data standards are adopted, the ONS data standard shall include a link to the standard where this is publicly available. In other cases, the ONS data standard will provide a summary of the industry standard (paying heed to copyright restrictions), with full guidelines on how to follow the standard.

Data standards shall include guidance as to their use. This guidance may be contained within the Data Standards Document or may be a separate document, depending on the complexity.

A backlog of data standards that have been requested but not yet developed shall be maintained, along with their priority for development.

Data standards shall be formally approved before being made available for use. Draft data standards are reviewed by the Data Architecture Review Panel (DARP), which includes stakeholders, SMEs and data architects as required. The Lead Data Architect is responsible for approving the data standards, and this approval shall be recorded in the data standards document.

All approved data standards shall be recorded in the Data Standards Catalogue. Both the Data Standards Catalogue and the standards themselves shall be centrally available. The Data Standards Data Architect shall ensure that new standards are added to the Data Standards Catalogue following their approval.

All approved data standards shall be reviewed at agreed intervals by the data standards data architect to ensure that they are current and relevant to the ONS.

Implementing data standards

Validation and verification of data standards compliance is the responsibility of the team processing the data, using a self-assessment checklist as agreed with the data standards data architect. The checklist shall be maintained by the data standards data architect, who will ensure that it is updated with new data standards as appropriate.

The data standards data architect is responsible for carrying out spot checks on the self-assessments, checking that evidence has been provided and testing the governance process.

If a team has problems conforming to a data standard, a concession can be requested. Applications for concessions shall be submitted to the Data Architecture Branch in conformance with agreed and published procedures. If concessions are granted, they shall usually be time-bound, with an agreed review date. On rare occasions, a permanent concession may be granted. The data standards data architect shall maintain a register of concessions and ensure that they are reviewed in a timely manner.

Data Standards Lead (Data Standards Data Architect) 

The Data Standards Lead ensures the consistent application of this policy to all ONS staff and is accountable to the Lead Data Architect.

In consultation with relevant subject matter expert(s), data architects and stakeholders, the Data Standards Lead:

• writes new data standards

• updates existing data standards

• reviews data standards on or before their review date

The Data Standards Lead also:

• arranges Data Architecture Review Panel meetings (DARPs) for formal review and sign-off of data standards

• considers requests for new data standards and approves or denies the request

• maintains the data standards backlog

• maintains the data standards catalogue

• carries out spot checks on self-assessments carried out to check compliance with data standards

• considers requests for concessions and approves or denies the request

• maintains a register of concessions and ensures they are reviewed.

Lead Data Architect

The Lead Data Architect is accountable to the Chief Data Architect and:

• approves (or rejects) data standards following review by stakeholders at the Data Architecture Review Panel meeting (DARP)

• approves (or denies) requests for concessions which have been escalated following denial for the concession by the Data Standards Lead

• approves (or denies) requests for new data standards and amendments to existing data standards which have been escalated following denial for the new or amended data standard by the Data Standards Lead

Delivery Managers

Delivery managers are accountable to the Project Lead. They:

• ensure that all systems for which they are responsible implement appropriate data standards

• apply to the Data Standards Lead for a concession where there is non-compliance with one or more data standards

• if a concession is not granted, optionally escalate this to the Lead Data Architect

Data Engineering and Operational Development Team

This team includes data engineers and data ingestion staff. They are accountable to the Deputy Director Data Engineering and Operational Development and are responsible for:

• applying data standards to all datasets

• carrying out self-assessments to record compliance with data standards, advising the delivery manager where there is non-compliance

Anyone processing data who creates new variables within a dataset

This group are responsible for:

• applying data standards to all datasets

• carrying out self-assessments to record compliance with data standards, advising the delivery manager where there is non-compliance