Data archiving policy

The Data Archiving policy sets out clear principles for the management of statistical data to data archives within the Office for National Statistics (ONS).

Data can be archived internally and managed by the ONS, or archived externally, such as at The National Archives (TNA).

The policy applies to all ONS employees, including staff on fixed-term, temporary or permanent contracts, staff on secondment, students, and contractors.

The Office for National Statistics' (ONS) data can be held online or offline. Usually data will be available online in live systems for production and/or research purposes. When data is no longer required to be held online (for example, because the speed of access to those datasets is not a critical factor), they can be held in offline data archives.

This policy is to ensure that data archiving internally within ONS or to external archives is considered to proactively preserve data assets for the long-term, remove the need for frequent database backups, and improve processing speeds on live systems.

Retention periods must be recorded in the ONS Information and Data Asset Register to ensure regular retention reviews for all data recorded on live systems and internal ONS archives are carried out as part of data governance activities.

Data archived internally within ONS must be supported by a Digital Preservation and Digital Continuity process.

This policy supports the Data Principles which underpin the Data Strategy.

This Data Archiving Policy applies to statistical data generated by the Office for National Statistics (ONS) as part of the statistical production process. The policy also applies to data that has been sourced or acquired for production and/or research purposes.

Data archiving ensures ongoing availability of data for future research purposes. Data will be archived internally at ONS and assessed for suitability for permanent preservation at The National Archives (TNA) 20 years following the creation date of the most recent dataset.

Personal data is retained in accordance with the UK General Data Protection Regulation (UK GDPR) and the Data Protection Act (DPA) 2018.

This Data Archiving Policy applies to all the data that the Office for National Statistics (ONS) holds to undertake its statutory functions as a Data Controller. This includes statistical data and any supporting metadata, for example data obtained through mandatory and compulsory surveys, or data obtained from administrative sources in the public or private sectors.

Data archives need to be managed in a consistent way across ONS. For example, the data should be managed and protected while held in data archives, and associated metadata should be maintained to support the management of data archives and context. Digital preservation and digital continuity measures should be adopted to ensure sustained access and usability over time.

Data and metadata should be archived to the ONS internal archive when they no longer need to be held online, but still need to be held for future research purposes, or because of a lengthy retention period. Retention of statistical data will be managed according to the Statistical Data Retention and Disposal Policy. Managing archives centrally will leverage reuse and reduce data duplication.

Datasets retrieved from the internal ONS archive will be accompanied by a manifest file. This supplies metadata regarding the datasets which are being retrieved, and which complies with the manifest file data standards.

Archived datasets that are no longer required, that are not worthy of permanent preservation at The National Archives (TNA), or cannot be held long-term for legislative purposes, will be disposed of securely in accordance with ONS policy and best practice. Disposal of statistical data will be managed according to the Statistical Data Retention and Disposal Policy.

Access to archived data needs to be authorised and approved (taking into consideration factors such as the regulatory and legislative constraints, the user role, and the sensitivity of the archived data).

Access should be provided in a timely fashion in accordance with service level agreements. Access should be managed at a level of granularity which allows administrators to prevent unauthorised access to archived data.

Data quality is governed across the entire lifecycle, up to and including data archiving and disposal, to ensure it is appropriately managed. Quality of archived data is managed according to the Data Quality Management Policy.

The Information Asset Owner (IAO) and Information Asset Liaison Officer (IALO) role holders are accountable and responsible for data governance activities assigned to them as part of their appointment to the role. Data governance duties relating to this policy include:

• the management of data assets throughout the data lifecycle and ensuring related governance metadata is recorded, accurate and complete

• the undertaking reviews of metadata when prompted, and decision-making for ongoing access to the data when retention periods are reached

• ensuring a data sensitivity assessment has been undertaken for assigned data assets and that associated risks are managed accordingly

• responsibility for decisions on use, transfer and access requests for data assets, and oversight around associated processing activities

• responsibility or accountability for risks associated with the data assets

• decision-making in relation to project or user accreditation relating to access to data

The Knowledge and Information Management (KIM) Team will approve user access to data held in the data archives after consulting with the IAO.

The KIM Team is responsible for ensuring that the governance metadata relating to data assets is accurate on the asset register and associated catalogues.

The KIM Team is responsible for supporting business areas with lifecycle management of information and records and own the interface with The National Archives (TNA) for the permanent preservation of selected public records, which includes datasets.