Discover data in the Integrated Data Service

All datasets currently available in the Secure Research Service (SRS) will be made available in the IDS as the transition progresses. As the IDS develops, several new, linked data assets will be made available through the service, offering a wider range of resources with which users can conduct their analysis.

The transition of data will be prioritised on frequency of usage. Data not in use will not transition into the IDS.

Data submitted to IDS will need to comply with published data standards and be accompanied by metadata. Any data transitioned from SRS to IDS will also require the relevant, complete metadata.

The standards will define mandatory, conditional and optional metadata and this will be checked at the point of ingestion to the platform. The quality of data and metadata will be monitored and reviewed on the platform, and there will be capability for users to raise queries and/or report errors against the data where required.

A partial data catalogue can be viewed on our public-facing IDS website. To access the full data catalogue, users must first become an accredited researcher and have access to the system. The full data catalogue is updated on a regular basis, whenever new datasets are acquired.

Plans are underway to introduce a full metadata catalogue that is publicly available, which will include a more detailed information about each dataset.

During the application process, researchers will need to specify the datasets that they require for their research.

We have an agreement in principle with Government departments who are keen to work with the IDS and share their data. We are working with those data owners to test and develop streamlined approaches for how data could be shared and accessed.

No data will be transitioned or made available through the IDS without advance permission from the data owner.

As an important strategic partner of the Office for National Statistics (ONS), Administrative Data Research UK will continue to fund the creation of new, linked administrative datasets and research using these resources through the transition from the SRS to the IDS and beyond.

Engagement with all data owners is still ongoing to discuss data sharing agreements for the IDS. The IDS will provide a central service where researchers can access data from a range of sources across government and beyond. Our aim is to minimise duplicative data sharing across organisations.

The design of IDS will make the linking of administrative datasets or creating matched comparison groups easier.

We will be relying on the Digital Economy Act (DEA) for most of the data sharing via the IDS. Within the research strand of the DEA, there is an exception for health data to be shared and we will discuss with NHS England about the use of alternative legislation. To facilitate the sharing of health data, the Statistics and Registration Service Act (SRSA) has already been used with the SRS. We will follow that approach in the absence of further legislative change.

Individual level data (raw data) will be accessible through Big Query. How it is accessed will depend on the complexity of the data format. If the dataset is small and not using SQL, tools are available in R and Python that can load them into memory as raw data file. With other data sets, SQL may be required to load those into memory. When they are loaded into memory in a notebook or session, users will have access to the individual-level data they have obtained. In their project, the user will have access to the de-identified data rather than the raw data with all of the identifiers included.

Any new data to be added to the IDS must be scheduled for ingestion through the relevant teams within the programme. Work is underway to design the ingestion processes.

If your organisation holds data that you would like to make available for research through the IDS, then please contact ids.acquire.service@ons.gov.uk and they will guide you through the process.

If you are a user and would like to request new dataset(s) for the IDS, please contact ids.acquire.service@ons.gov.uk to discuss your requirements.

The facility to upload a researcher’s own data to IDS is being explored as part of the development of the platform.

If the information you are looking for does not already exist on these pages, you can contact the SRS Customer Support team for more information by email at srs.customer.support@ons.gov.uk.