Source: Radio New Zealand
Statistics Minister Shane Reti has called the national system “dated and constrained”. RNZ / Samuel Rillstone
A unique trove of information for measuring the impact of government services on millions of New Zealanders is sitting on “old and increasingly unstable” technology. It is hard to use and badly in need of an upgrade – but the Integrated Data Infrastructure system – run by Stats NZ – is also crucial to the government’s overarching social investment approach because it gives answers that cannot be found anywhere else – as long as you can coax them out of it.
The message read like the sort of warning that is sent when e-mail storage runs low: “We currently have 1% capacity, and this is impacting all users.”
The e-mail – sent by Stats NZ last Wednesday – was talking about its main “sandpit” – or testbed for researchers – inside a system so valuable anybody who wants to use it has to sign up to a lifetime secrecy pledge.
Stats NZ told the researchers to dump their old data tables in the Integrated Data Infrastructure system (IDI), adding it was going on a “clean-up” of old accounts because its budget stretched to 550 researchers, not the 1000 who have signed on.
Statistics Minister Shane Reti has called the national system “dated and constrained”, and the situation is especially fraught when the government is relying on evidence-driven changes to social programmes.
“The IDI was developed in 2011 and is still in prototype form,” Stats NZ told Treasury late last year.
“Its capacity has been exceeded and it is not future-proofed to handle the increase in demand for person-level data and analytics.”
Another briefing to Reti in July 2025, newly released to RNZ, said: “While the IDI is a critical tool to help accelerate social investment, its ageing infrastructure and complex user experience need to be upgraded before it is ready to support the Social Investment Fund.”
It was “clunky and slow”, with high technical barriers, the minister was told.
‘Incredibly difficult for them to actually find useful information’
Stats NZ has also told ministers it had made some effort to improve the vital system over the years.
But this was clearly not enough and a lot more was needed, according to the Reti briefings.
He has told his officials to move “at pace” on an overhaul, he told RNZ.
The officials had got as far as a preliminary business case going before Cabinet a few weeks ago.
Meantime, immediate work was continuing to make sure the system was usable and secure, Reti said.
The task is daunting – while Stats NZ is already working on a new tech platform for the IDI, there are 15 billion rows of data, which can not be found anywhere else.
Finding it at all within the datasets could be inordinately difficult, according to Gisborne researcher Malcolm Mersham.
“It is pretty clunky and very challenging to find the data,” Mersham said.
Mersham is research and insights leader at Trust Tairāwhiti in Gisborne, which wanted to mine loads of data – including health, economic and household stats – to set a baseline measure of the happiness of the locals.
“The reason we went to the IDI is because publicly that information wasn’t available for us in Tai Rāwhiti,” with data typically lumped in with Hawke’s Bay.
However, they struggled to get into the IDI at all.
“There’s only really a few that can sort of navigate the research project piece.”
They needed expert help to apply, then more expertise from someone familiar with the labyrinth – and who knew some coding to unlock its secrets.
“For any person from the streets or average Joe, it would be incredibly difficult for them to actually find useful information about their communities,” Mersham said.
Ultimately they did not find what they wanted.
“We’re poor, we’re Māori, we’ve got a lot of deficit kind of data … So, this is like if you get a traffic fine, it’ll pop up.
“The [government administrative] admin data is really sort of output of activity that government does to you, if that makes sense.”
They instead needed measures of whether people were happy and connected.
While their IDI project did not get there, it pointed them in the right direction – creating their own regional well-being survey, which is now in its fourth year, Mersham said.
They have found high levels of happiness and great connectedness, even where incomes are low.
“We would love to be able to publish that into the IDI.
“I think the future of the IDI, particularly with these upgrades, is what I’d like to see is the ability… to absorb datasets outside of just the admin data from government.”
Capacity pressure outside sandpits
At Auckland University, social sciences professor Barry Milne received the Stats NZ ‘1 percent capacity’ email while he was using the IDI on a project that looked at the impact of acquired brain injury on mental health.
“Once you get used to it, I think it’s easy to use,” Milne told RNZ.
Barry Milne, Professor in the School of Social Sciences at the University of Auckland and director of the Compass Research Centre. Supplied
But he said it was “creaky … I get the sense it’s kind of just being patched up a little bit to keep up with the demand for it”.
Parts of the IDI got a $1.4m patch-up in the eight months to June 2025, somewhat improving access.
“My sense is it needs a kind of complete restructure,” Milne said.
Government agencies have used the IDI in simulations to research policy change and monitor outcomes, such as Treasury’s tax and welfare model; MSD’s social outcomes modelling; and Oranga Tamariki’s children’s well-being model. Other projects have assessed how women’s pay is impacted by taking time off to raise children and men’s pay if a prostate cancer diagnosis comes late.
“It’s still really, really good,” Milne said.
It was the envy of research colleagues overseas, who could not believe he paid just $500 for access, and who often had even clunkier data systems to deal with, he added.
Stats NZ said the $1.4m built a sandpit just for Social Investment Agency researchers, taking the pressure off other shared space.
In June it also set up two additional sandpits for high demand users.
The real capacity pressure was on the infrastructure and data processing, despite “significant improvements and efficiency gains” in recent years, it said.
“We are limited to updating the IDI three times a year, and we have a limited number of new datasets that can be added,” Stats NZ said.
“This is one of the many challenges we would seek to address with investment in infrastructure upgrades.”
Risks in the fix
But Stats NZ has a long way to go to fix what it called a “world-leading system” and the country’s “only source of high-quality, de-identified, integrated data”.
The briefings showed the data being put into it from other agencies was “poor quality, not in a standardised format and missing in critical areas”.
But Professor Tahu Kukutai of Te Ngira Institute for Population Research said fixing that could be hard, because these other agencies might not help enough.
She said another question was around how much public trust existed in the government’s recent move to rely more on ‘administrative data’ – which has put an end to Censuses as we know them – and records its interactions with individual citizens, for everything from school and GP enrolment to speeding tickets.
“Māori trust in Stats NZ isn’t great,” she said, after a “failure” to implement Māori data governance and other measures.
‘The IDI will remain secure’
Treasury told ministers in late 2024 the IDI was an example of “inflexible and difficult to use infrastructure which cause inefficiencies and security risk within their system”.
At the public face, the assessments can be rosier.
The Social Investment Agency (SIA) – set up last year with former police commissioner Andrew Coster at its head – said: “The IDI is a world-leading resource with a great breadth and depth of data. Researchers can use it to analyse populations and investigate the impact of services and programmes on people’s lives.”
More than most agencies, the SIS needs the IDI to exhibit reliability.
But Reti told RNZ the data system was old so its “processes for data integration and analysis are slow and require manual support”.
“Stats NZ have assured me that with the immediate work underway using time-limited funding, researchers and decision-makers will continue to access the data they need and the IDI will remain secure.
“However, significant work is required to deliver the infrastructure upgrades needed to ensure our data infrastructure can scale-up in the long term.
“This is a complex undertaking, but I have made my expectation clear to officials that this project is to continue at pace and a long-term solution must be in place to support the government’s programme of work,” Reti said.
Mersham was hoping the officials will ask him what the revamped system should look like.
For those based in Gisborne, one physical barrier was that researchers could only use the IDI in a Stats NZ-run data lab.
The barriers needed to drop, Mersham said, and official papers also mentioned this.
“But I also appreciate it is a big machine,” he said. “I can totally appreciate where they find themselves right now.”
Sign up for Ngā Pitopito Kōrero, a daily newsletter curated by our editors and delivered straight to your inbox every weekday.
– Published by EveningReport.nz and AsiaPacificReport.nz, see: MIL OSI in partnership with Radio New Zealand