"Reckless" doesn't begin to describe the new culture at the top of the Bureau of Statistics.
Its most important product apart from the census is the monthly employment survey. Two years ago it decided to modernise. It moved much of it online, accepted a lower response rate and changed the months in which different questions were asked.
Against the advice of ABS veterans, it didn't run a backup survey using the old system.
In August 2014 the number of Australians officially employed jumped an incredible 121,000. The next month, September, it dived 172,000, or it would have had the ABS not pleaded with users to ignore the seasonally adjusted numbers which it no longer trusted.
Without a backup – a parallel employment survey conducted under the old system – it was impossible to tell what was wrong and what was right....
With no leader, treasurer Joe Hockey and prime minister Tony Abbott had left the position at the top unfilled for the best part of a year, the ABS developed grander plans.
It wanted to abandon the 2016 census altogether, moving from five-yearly to 10-yearly, to save $200 million.
Told by the government that would require legislation that would be unlikely to get through the Senate, it decided to get extra value by using the names collected with the census to create permanent linkage keys. Previously destroyed after processing, the names would be kept indefinitely in a separate file and used to link the answers to census questions to the answers to other ABS surveys and data collected from organisations such as the Pharmaceutical Benefits Scheme and the Australian Tax Office. If it could find out whether Australians or certain ethnic backgrounds or family circumstances were more likely to claim certain tax deductions or be prescribed certain medicines it could sell the results.
The only problem was that it had put up the idea before to an independent privacy impact assessment, which had savaged it. So it conducted its own assessment in house, which found no problem. It published the results quietly late last year (there was nothing about in in the mainstream press) and then on the Friday before Christmas announced it would retain every name indefinitely in a press release with the misleadingly bland title of: ABS response to Privacy Impact Assessment.
By April it had backed down. It was going to keep the names for only four years, but it would keep the linkage keys created from them for as long as would be needed, which could be forever.
Woefully unprepared to explain why it now wanted to retain names, it emphasised instead the digital nature of the census. It would save millions by posting login codes to most of the population rather than delivering forms. Had it delivered, or even posted, forms it would have had a backup.
Instead it gave most of Australia only one way to submit census forms, emphasised the importance of the survey it had previously tried to ditch, threatened fines of $180 per day for people who didn't comply, and underestimated either the strain on the system or the security of the system.
At almost every step of the way the government has been hands off. The latest minister (Michael McCormack has been in charge of the ABS for less than two weeks) gives the impression the decision to retain names didn't even go to cabinet.
There's already been an inquiry into the ABS. It found it was not properly ready "to maximise the value of all government-held information". Which might be part of the problem. Until now one of Australia's most trusted organisations, it has tried to catch up too fast. The government has looked the other way.
In The Age and Sydney Morning Herald