How government can capitalise on a revolution in data sharing

A watershed moment in the culture of data sharing, the pandemic has led to the use of linked data increasingly becoming standard practice. From linking census and NHS data to track the virus’s impact among minority ethnic groups, to the linking of timely local data sources to support local authorities’ responses, the value of sharing data across boundaries was self-evident. 

Using data to inform a multidisciplinary pandemic response accelerated our longstanding work on data capability. To continue this progress, there is now a need to make government data more organised, easier to access, and integrated for use. Our learning has guided the development of a new cloud-based platform that will ensure that anonymised data about our society and economy are now linked and accessible for vital research and decision-making in the UK.

The idea of sharing data to maximise impact isn’t new to us at the ONS – we’ve been doing this successfully for over 15 years through our well-respected Secure Research Service (SRS). The new Integrated Data Service (IDS) is the next step in this data-sharing journey, where, in a far more advanced form, government will have the ability to work with data at source – in a safe and secure environment – rather than moving data around, which currently creates friction and significant cost. The service, being compliant with the Digital Economy Act, opens up opportunities to capitalise on the often-underutilised research elements of that key legislation.

The launch of the full IDS in the spring of 2023 will see ready-to-use datasets made available to cross-government teams and wider research communities, enabling them to securely share, link and access them for vital research. The service is a collaboration among institutions to work on projects that shed light on some of the big challenges of the day, and to provide the ability to answer questions that we don’t yet know we need to answer.

Our longstanding partnership with Administrative Data Research UK has helped us to shape the development of the IDS by putting the wider research communities’ current and emerging needs at the heart of the service.

The idea of collecting data once, and it being used many times, might sound like common sense, but the reality is that datasets have tended to remain where they were gathered and used for just one single project. Until recently, analysts working with government data have had to navigate complex legal frameworks and isolated working arrangements that delay, or in some cases prohibit, important and innovative research.

Data-driven government is not a new concept, but the challenge we have had to overcome has been to find a solution that allows us to share timely and relevant data securely across government, academia and the wider research community to support effective analyses and decision-making. The IDS provides a platform that can meet this need, while maintaining high standards of data security and governance.

It is tricky for people to picture how the complete system will look during this beta phase, but the platform’s development has been characterised by a movement toward openness. The IDS will replace the current SRS, which began as a very closed system that saw researchers working in locked rooms under strict surveillance. While the government model places a protective bubble around data assets held within its departments, we have made significant progress to provide as much openness as possible while maintaining the highest standard of ethics and security.

Although analysts recognise the benefits of the system, we must continue efforts to get more data providers to sign up. In essence, we need more data and we need more users. We have plans in place to increase both. Crucially, the cultural shift that has been required to see departments share their data, in recognition of the benefits for wider analysis, is well under way.

Throughout the IDS’s beta phase, a growing range of integrated data assets have made new analysis possible, providing otherwise untapped insight on the key challenges we face, from economic recovery and public health to regional growth and climate change. An ONS and HM Treasury collaboration is investigating how wages change across the country, to find new ways of ensuring that no communities are left behind. Elsewhere, our work with the Department for Business, Energy and Industrial Strategy is analysing how text from local BBC news sources can be used to understand the concerns of communities across the UK.

As for our understanding of the economy, VAT returns data has transformed the level of detail we produce on GDP, and since 2019 we have published quarterly data on regions, local authorities and specific sectors, pinpointing the driving forces behind national changes.

Data now accessible from HM Revenue and Customs has provided us with more detail on the numbers of people on the payroll and how much they earned, with breakdowns across industry verticals, regions and age groups. This data helped us to paint a closer-to-real-time picture of employment in the UK and put a number on the impact the pandemic was having.

And as we measure the UK’s progress toward reaching net zero, our work with the Valuation Office Agency has advanced the information we hold on the energy efficiency of homes around the country.

As we progress with the IDS roll-out, our focus is to act nimbly, securing the best-quality data in the right way, combining it with administrative and alternative data assets and then making it available for wider analyses and insight. Making data readily available to analysts to help answer complex policy questions is essential for addressing the cross-cutting and complex issues required to improve today’s society.  

Alison Pritchard is deputy national statistician and data capability lead at the Office for National Statistics.