Thursday, November 03, 2016

Productivity Commission: how big data could work for us

What if we were on the cusp of one of the biggest ever advances in productivity and we didn't recognise it?

That's how it must have been for Alexander Fleming with the discovery of penicillin, for university technicians with the development of the internet, and for Bill Clinton, who with the stroke of a pen in the year 2000, made highly accurate military global positioning satellites available for everyone to use for free.

Peter Harris believes we are on the cusp of another transformation about as big – one only made possible by the development of the internet and all the things that surround it.

It's the exploitation of data. On one estimate we are now generating as much digital data every two days (five exabytes) as we generated in an entire year at the start of the 2000s.

Some of it is cat videos. Much of it mundane. But an awful lot is useful, and his best guess is that only 5 per cent of the useful stuff is being used, a figure that puts us way behind the countries we usually like to compare ourselves to, especially Britain and New Zealand.

We are behind partly for privacy reasons, partly because potential users don't know what data government agencies hold, and partly because the machines that hold it often can't talk to each other, even within the same hospitals.

It is an outrage that sick patients still have to act as information conduits between healthcare providers (10 to 25 per cent of the medical tests ordered are thought to be duplicates) and a disgrace that 60 years after the Thalidomide tragedy we still don't link prescription data to hospitalisation records to get insights into the side effects of drugs.

Research that could have saved the lives of Indigenous women was delayed five years while the researchers waited for ethics approval to see cervical cancer screening data; researchers wanting to study the link between vaccination and admission to hospital have had to wait eight years and counting.

Harris runs the Productivity Commission. It is a measure of his belief in the importance of the data inquiry commissioned by the Turnbull government that he decided to chair it himself and personally briefed journalists on the contents of his draft report on Wednesday.

His first recommendation is that all government-funded entities create easy-to-access registers of everything they've got. He wants them published by October 2017. If anyone wants a machine-readable copy of something on a register, they should be able to get it for free or for marginal cost, unless there are powerful reasons for holding it back.

Given how much personal data so many of us willingly or carelessly give away every day, he isn't particularly concerned about the privacy risks of releasing de-identified personal data (and allowing it to be linked to other data, as the Bureau of Statistics wants to do with the census), saying the risks are "likely very small". Where there's a clear public interest, he wants researchers to be given access to private information in secure rooms.

Right now they are often required to destroy datasets they create in medical and other research, a practice he says is akin to "book burning". He would require them to keep it.

Really important information would be curated in "national interest datasets", overseen by a national data custodian who would report to the parliament.

But that's just half of it. Right now, in spite of a widespread belief to the contrary, you and I don't have access to our own data.

If I ask my music streaming service for details of my listening habits, or my search engine for details of my search history, or my insurer for details of my claiming history, or my supermarket for details of my shopping history, or my electricity supplier for details of my usage history, they are perfectly entitled to refuse to hand them over. I might want to take them to a competitor.

Harris wants to enshrine in law my right to take them to a competitor. Even better, he wants my providers to hand them to the competitors or brokers I select at my direction. I probably wouldn't be able to make much sense of a machine-readable account of my electricity use, but a competitor would.

Suddenly, competition could really work. And it would cost almost nothing. There would be no privacy concerns because it could be released only at my direction. Harris would also give me the right to request edits or corrections to the data firms have on me, to be informed about their intentions to sell or pass it on, to be able to order them to stop collecting it (at the risk of losing the service) and to appeal automated decisions that deny me services or charge me more on the basis of it.

He is talking about a revolution. It's a revolution we ought to embrace and direct, rather than sit back and watch.

In The Age and Sydney Morning Herald