Showing posts with label abs. Show all posts
Showing posts with label abs. Show all posts

Tuesday, April 02, 2024

From where we work to what we spend, the ABS knows more about us than ever before: here’s what’s changing

How much were prices rising in January when Shadow Treasurer Angus Taylor said inflation was “rampant”?

The prices that give us a good steer on inflation were falling, by 0.4%.

That’s the change that month in what the Bureau of Statistics calls the consumer price index “excluding volatile items”. The items it excludes (because they are often affected by supply disruptions) are fruit, vegetables and fuel.

Apart from that, the measure of prices I just quoted is the best monthly measure of the prices of everything that households buy in the proportions they buy them.

Not all prices were falling. The price of alcohol was up, the price of bread was down, the price of rent was up, and the price of tourist accommodation was down. It’s only on balance (excluding volatile items) that prices fell.

This is the sort of thing we wouldn’t have known about until just a few years ago. Up until late 2022, the consumer price index was calculated only four times a year, and even that was a herculean feat.

A ten-fold increase in data

The bureau had to collect what used to be 100,000 separate prices for each of those four surveys – a huge number collected in person, either over the phone (“hello, can you tell me your current price for…”) or in stores via handheld devices.

The cost to the bureau, and the number of staff involved, was enormous – big enough to make a monthly measure impossible, as important as that would have been to a Reserve Bank that set interest rates monthly and needed a monthly read on inflation.

But in the last few years the use of supermarket scanner data, “web scrapping” to collect online prices, and data feeds direct from the computers of rental agents and all sorts of other businesses have cut costs enormously and increased the number of prices collected each quarter almost ten-fold to 900,000.

The bureau says the monthly index isn’t as comprehensive as the quarterly index yet, but it will be by the end of 2025, at which time the bureau will use it to replace the quarterly index, delivering something of the same quality 12 times a year.

That’s just one of the ways in which an explosion of previously-inaccessible data is transforming the way the bureau goes about its job and is set to make statistics that used to be only fairly reliable suddenly very reliable.

Retail figures set for the chop

For more than half a century, every month since April 1961, the bureau has published an update on retail spending – how much we are spending in shops.

The survey used to be quite useful. Back when it started, we did more than half our spending in shops. These days it’s only one third, the rest is on services.

And the retail survey was always a pretty rough-and-ready way to find out what we spent in shops. Each month the bureau surveys about 700 large businesses and 2,700 smaller businesses selected at random. It uses phone calls and paper forms.

Meantime, in part due to the national emergency created by COVID, it’s been given access to something better. Australia’s big four banks agreed to give the bureau de-identified card and transaction data to enable it to quickly get a handle on how much we were spending early in the pandemic, and they’ve kept providing it.

It turns out to be very good indeed. It covers far more retail outlets than the retail survey ever did, as well as spending on services and spending overseas, and it divides spending into categories based on the type of merchant.

It doesn’t directly cover what we spend in cash, but there’s a lot less of that than there used to be. It’ll replace the retail survey from the middle of next year.

Millions instead of thousands

The mammoth monthly employment survey of 24,000 households remains in place, as do the doorknocks that begin each household’s eight-month turn at completing the survey, but alongside it the bureau is developing a far more comprehensive measure using payroll data submitted to the tax office.

While payroll numbers can’t tell us everything the employment survey does (they can’t yet tell us the hours people work and whether are looking for work) they cover millions of Australians instead of thousands, and come out weekly.

The bureau is doing the same sort of thing almost everywhere. For more than a century it has surveyed farmers to find out what they are growing. It’s begun supplementing that with data from satellites and the machines used on farms.

The ultimate goal of all of these changes, gathered together under the banner “Big Data, Timely Insights” is to ask as few questions as possible. Why run a survey, when you can find out directly?

Big data, timely insights

It’s far harder than it looks. A lot of the so-called administrative data provided to banks and other organisations isn’t sorted in a way that makes it useful. That’s where the bureau is concentrating its efforts. The more it succeeds, the less it will need to bother us and the better the information it will produce.

In the meantime, here’s an update on those inflation figures, the ones that come out monthly. In February, the consumer price index excluding volatile items did not change, meaning in that particular month, inflation was zero.

Even better, adding the past six months together (and multiplying by two) gives you an annual inflation rate of 2.5% – slap bang in the middle of the Reserve Bank’s 2-3% target band, suggesting things are moving in the right direction.

It’s too early to declare victory over inflation that was at one stage heading towards 8%, but at the moment the monthly figures show things heading down.

If the direction changes, the bureau will tell us, quick smart.The Conversation

This article is republished from The Conversation under a Creative Commons license. Read the original article.

Read more >>

Wednesday, December 01, 2021

GDP went backwards, but look at what’s to come

The most revealing graph presented in Wednesday’s September quarter national accounts is one showing what has happened just beyond the end of the September quarter, in the one we are in now.

Melbourne’s lockdown ended on October 27.

The graph uses anonymised bank account data to show what happened to spending in Victoria as soon as the lockdown was lifted.


Selected Victorian spending data

Aggregated bank data. Index for May 2020 = 100. ABS

Spending on clothing, furnishings, recreation, transport and restaurants and hotels surged.

As happened after last year’s lockdowns, Victorians returned to spending pretty much what they had before.

The September quarter national accounts released on Wednesday are a statement of their time – they show what things were like when NSW, Victoria and the ACT were locked down.

Australia’s gross domestic product shrank 1.9% in the three months to September, after climbing for four consecutive quarters following the record hit of 6.8% from the first wave of COVID and last year’s lockdowns.


Australian quarterly gross domestic product

Chain volume measures, seasonally adjusted. ABS

The biggest hit to GDP came from household spending, down 4.8% in the quarter.

National spending on hotels, cafes and restaurants fell 21.2%, spending on recreation and culture fell 11.8%, and spending on transport fell 40.8%.


Household final consumption expenditure

Chain volume measures, seasonally adjusted. ABS

But the decline was anything but national.

Whereas spending in hotels, cafes and restaurants collapsed 33% (or more) in each of the states that were locked down, in the states that weren’t, it barely suffered.

Aggregate spending shrank 6.5% in NSW, 1.4% in Victoria and 1.6% in the ACT, while climbing strongly in the states that weren’t locked down, surging an impressive 4% in the Northern Territory and 4.2% Tasmania.


State final demand, September quarter

Seasonally adjusted. ABS

Nationally, personal saving soared, with the jump centred in the lockdown states as those households whose income hadn’t taken a hit saved more because of concern about the future and fewer opportunities to spend.

The national household saving rate bounded back up to an extraordinary 19.8% of household income from the 11.8% it fell to in March, after hitting an all-time high of 23.3% in the first wave of lockdowns.


Household saving ratio

Ratio of saving to net-of-tax income, seasonally adjusted. ABS

The bank-sourced data on post-lockdown spending in the lockdown states suggests household saving is already on the way down.

At his parliament house press conference, Treasurer Josh Frydenberg spoke of the unusually high saving rate as a source of future spending.

“Not all of it is going to be spent,” he said. “But it’s a lot of damn money that’s been accumulated”.

If household spending had been the only thing driving changes in gross domestic product, it would have been down 2.5%. Working in the other direction this quarter was a jump in net exports and a jump in government spending. Neither business investment nor housing construction changed much.

Organisation for Economic Co-operation and Development forecasts released late Wednesday have the Australian economy growing 4.1% in 2022, up from 3.8% this year, slipping back to 3% in 2023.

Sustained low unemployment forecast

The OECD Economic Outlook has Australia’s unemployment rate falling to 4.7% in 2022 and 4.3% in 2023. It says as inroads are made into unemployment, wage and price pressures will build, but are expected to “remain contained”.

The organisation welcomes Australia’s commitment to net zero emissions by 2050, but says the strategy should be “comprehensive, effective and inclusive”. Its report backs a call for an independent review of the Reserve Bank.

It forecasts global growth of 4.5% and 3.2% in 2022 and 2023.

Peter Martin, Visiting Fellow, Crawford School of Public Policy, Australian National University

This article is republished from The Conversation under a Creative Commons license. Read the original article.

Read more >>

GDP is like a heart rate monitor: it tells us about life, but not our lives

How much cash would you need to be paid to agree to live without a smartphone for a year?

If you are like the typical American, the answer is US$10,000 – which is far, far more than what we are actually charged for having and using smartphones.

How much would you need to be paid to live without a computer?

According to the same research, just published by Stanford University’s Hoover Institution, a typical American would want US$25,000 to live computer-free for a year.

For the GPS system that lets us map where we are on all our devices, the answer is US$3,000; for streaming services such as Netflix the answer is another US$3,000.

For refrigeration the answer is US$10,000; for air conditioning, another US$10,000; and for running water US$50,000.

The point of this study, by economist Tim Kane, is that if we add up the worth to us of everything the economy produces each year, we get much, much more than the gross domestic product – even though GDP is meant to be a summation of the prices paid each year.

Not a day goes by when we don’t get astounding value for money: on Kane’s estimate, about 20 times what we pay.

GDP monitors changes, not our lives

It’s a useful perspective to bear in mind ahead of the latest Australian gross domestic product figures, being released on Wednesday.

Those figures will show Australia spent less, earned less and produced less in the lockdown-affected September quarter months of July, August and September than in the three months before – about 3% less on private estimates.

It won’t be a “recession” because in Australia that’s generally taken to mean two consecutive quarters of those things going backwards. And we already know spending, earning and production all started climbing as soon as the lockdowns ended at the beginning of the quarter we are in now.

The GDP has the same relationship to life as a heart rate monitor has to health.

There’s more to GDP than you might think

Behind the headline figure you hear about are actually three different measures.

GDP(P) is a measure of everything that’s produced in the quarter. The Bureau of Statistics has the unenviable job of adding up most things that are produced at market prices (and having a stab at trying to infer market prices where they are not apparent) in industries as diverse as mining, financial services and education.

It tries to count each thing only once, which is difficult because some things are used as inputs to others. Its work is made harder by relying partly on surveys and partly on complete sets of data from organisations such as the Tax Office.

Ask whether it uses guess work, you will be told it uses “informed judgement”.

GDP(E) is a totalling of government and household expenditure to buy those products. After adjusting for imports and exports it ought to equal GDP(P), but imperfections in measurement mean it usually doesn’t.

Then there’s GDP(I), which is a measure of the income households and businesses get from working and selling those products. Again, it ought to equal the other two, but it usually doesn’t.

After trying to get the three measures nearer each other (perhaps there was something somebody missed) the technicians in the bureau simply average the three, producing GDP(A). That’s what goes up on the ABS website at 11:30am AEDT Wednesday, followed by a Treasurer’s press conference and loads of analysis.

It needn’t indicate an underlying condition

Just as a heart rate monitor needn’t tell us much about health, because even in healthy people hearts beat slower while sleeping and faster while awake, GDP needn’t tell us that much about the condition of our lives.

A lot of the economy went to sleep during this year’s and last year’s lockdowns and is now waking up. The GDP will show that, but at least on Wednesday it won’t tell us more than that.

As it happens, economic growth has been weakening over time. Annual GDP growth is no longer the 3-4% it typically was between the early 1990s recession and the 2008 financial crisis. In the decade leading up to COVID it has been much lower, rarely touching 3%.


Annual financial year GDP growth

Financial year on financial year growth, 2002-03 to 2018-19. ABS

Put starkly, for little-understood reasons unrelated to quarterly fluctuations or COVID, we are getting better off more slowly than we were.

There are always people who say this doesn’t matter, we should be happy with what we had (and as I noted, much of what we’ve had isn’t counted in the GDP).

There is an underlying condition nonetheless

But it matters a good deal, because ever since economic growth took off in the 1870s we’ve grown used to things continually getting better, and have come to expect it.

US economic historian Brad Delong uses an 1880s science fiction book to illustrate how much we’ve come to regard improving living standards as a birthright.

In Looking Backward, Edward Bellamy purports to look back from the year 2000.

At one point a hostess asks if he would like to hear some music. Instead of playing the piano, she merely touched one or two screws and “immediately the room was filled with the music of a grand organ”, one of four she could dial up by landline.

It appeared to him that

if we could have devised an arrangement for providing everybody with music in their homes, perfect in quality, unlimited in quantity, suited to every mood, and beginning and ceasing at will, we should have considered the limit of human felicity already attained, and ceased to strive for further improvements.

He got it wrong.

Peter Martin, Visiting Fellow, Crawford School of Public Policy, Australian National University

This article is republished from The Conversation under a Creative Commons license. Read the original article.

Read more >>

Wednesday, August 25, 2021

The official figures say wages aren’t growing — here’s why they’re wrong

Have you heard about the latest wage figures? I hope not. They’re meaningless.

What the widely quoted measure of average weekly earnings purports to show is that wages grew a mere 0.1% over the year to May. It’s not true. It’s not what happened. For most of us, wages grew by much more.

That’s not to say wage growth has been high — the best estimate is that private sector wages have climbed 1.9% over the past year and public sector wages a record low 1.3% — but both are still well above nothing, and generally well above our near-record low rates of consumer price inflation.

A check-in with reality would tell you that mid last year the Fair Work Commission lifted award wages 1.75%. Mid this year it lifted them 2.5%.

So how could it be that the official figures, published by a trusted organisation, the Australian Bureau of Statistics, show average earnings static, climbing just 0.1%?

The first thing to say is that the bureau is probably embarrassed by the figures.

They are “not designed to produce movement in earnings data” it says on its website, before acknowledging that’s exactly what they are used for.

‘Not designed’ to measure wage growth

Australia’s pensions are adjusted twice a year in accordance with a formula that includes average weekly earnings.

The figure is built into private contracts. If it wasn’t published, many contracts wouldn’t work.

To create it, the bureau surveys about 5,130 employers every six months, asks what they are paying their workers, and uses the answers to calculate an average female wage, an average male wage, an average part-time wage, an average full-time wage, and a lot of other averages besides.

The ‘average wage’ isn’t typical

One problem is that averages are not representative. The survey suggests the average full-time wage is A$90,330, whereas in reality six in ten earn less.

The mid-way (median) full-time worker earns $10,000 less. The average is boosted by a few enormously high earners and can’t be taken seriously.

An entirely separate problem arises when you try to use averages to calculate growth. The average is only an average of what’s averaged, and that can change.

When low-wage workers lose jobs…

Here’s an example. What would happen if a recession caused everyone working only four hours per week to lose two hours? It would push their earnings down and push down average weekly earnings, which would be about right.

But what if each of those people lost a further two hours, taking their hours down to zero. Their low hours would no longer be included in the total to be averaged, and (without them in it) average earnings would climb.

…the average wage goes up

That’s what happened a bit over a year ago. The bureau says COVID restrictions “led to a large decrease in the number of jobs, people employed and hours worked, with lower-paid jobs and industries particularly impacted, including jobs in accommodation and food services, arts and recreation services”.

The loss of those lower-paid and low hours jobs in catering, the arts and other industries “had the effect of increasing the value of average weekly earnings”.

Layoffs pushed the average wage up.

Fortunately, the bureau says by November many of the low-wage workers laid off got some hours back, depressing growth in the average wage (but not growth in any actual wages) resulting in recorded growth of just 0.1% in the year to May.

Many have probably since lost hours with this year’s renewed lockdowns, pushing average wages (but not actual wages) higher again.

It’s enough to make you think the legislation and contracts should switch from a measure that’s close to worthless to one that actually measures wage growth.

The bureau offers such a measure. It’s called the wage price index, and the bureau has been trying to encourage people to switch to it since 1998.


Read more: Other Australians don't earn what you think. $59,538, is typical


It is also built around a survey of employers, but rather than asking how much they pay each worker, it asks how much they pay for each job title and classification. The bureau calculates growth by comparing like with like, regardless of how many people were employed in each classification at the time.


Wage Price Index

Annual growth in total hourly rates of pay excluding bonuses, public sector and private sector. ABS

The results are believable: private sector like-for-like wages climbed 1.9% over the past year, and public sector wages 1.3%.

But even they are not right when it comes to the wage growth of individuals.

Individuals get promoted, and (much less often) demoted. They change jobs, usually for better ones.

People aren’t positions

So if you were trying to use the recent like-for-like wage growth of around 2% per year as a guide to what will happen to your own wage (in order, for instance, to work out whether you could afford a mortgage) you would probably guess too low.

It’s why many Australians — those who’ve got not only regular pay rises but also promotions — wonder what the fuss about low wage growth is about.


Read more: Top economists say cutting immigration is no way to boost wages


A good measure of the actual wage growth of Australians doesn’t yet exist, although it might soon. The bureau is working on tracking individuals through the use of payroll data reported to the tax office.

In the meantime the (HILDA) Household, Income and Labour Dynamics in Australia survey that tracks 17,000 Australians over time finds that the actual wage growth of full-time workers is indeed higher than the like-for-like figure suggests (which might help explain soaring home prices) although it too is weakening.

Part time workers don’t seem to get the same benefit.

As Mark Wooden, director of the HILDA survey puts it, “Australians in full-time work are doing pretty well – provided they remain in employment”.The Conversation

Peter Martin, Visiting Fellow, Crawford School of Public Policy, Australian National University

This article is republished from The Conversation under a Creative Commons license. Read the original article.

Read more >>

Wednesday, March 10, 2021

How the ABS became our secret weapon

If we survive this economic crisis (and it is looking increasingly like we will, although the end of JobKeeper at the end of the month will be a setback) it will in large measure be because this time we’ve had a real-time picture on what’s been going on.

Last time, we were flying blind.

In what must have been one of the worst-timed decisions of an incoming government ever, in 2008 the newly-installed Rudd government slashed the budget of the Australian Bureau of Statistics ahead of the global financial crisis.

In its first budget it hacked A$28 million off ABS budget, and told it to work out what to cut.

The ABS lopped off its job vacancies survey, closing it down in May, just months before the collapse of Lehman Brothers in September.

Then in July it cut the size of its employment survey from 54,400 people to 41,100, making the results less accurate just as accuracy began to really matter.

Rudd and his staff had to navigate partially blindfolded.

It was, as the then head of the treasury’s macroeconomic group David Gruen said at the time, as if the Titanic was sailing into iceberg-infested waters while those with the requisite skills were hard at work “in a windowless cabin”.

Twelve years on — astoundingly — David Gruen has found himself on the other side of the cabin wall as head of the ABS.

He took up the job on December 11, 2019, just days after the first Wuhan resident fell sick with what turned out to be coronavirus.

By the end of February he had “this feeling I last had in the middle of 2008”.

Not much coronavirus had spread to Australia by that point, but as Gruen recounted to the Canberra branch of the Economic Society, it felt like “something big was coming”.

Something big was coming

Gruen called a brainstorming session and asked senior staff what data they could produce quickly — far more quickly than usual — that would tell people what was happening in near real-time.

The business conditions unit said it could run a survey of 1,200 businesses, but that it would cost money — $20,000. Gruen told them to spend it. The survey began on March 16, ran for three days and was published on March 26, a record-quick ten days after the first questions were asked.

That initial survey asked how COVID was hurting each business, what it expected. Then requests started coming in for further questions about cash on hand, revenue and employees. Month by month the survey evolved as the crisis evolved.


Read more: Australia's first service-sector recession unlike those that went before it


Then the household survey unit realised it could do one. It repurposed a panel it had assembled for a different survey and went back to the same households month after month for real-time insights into things such as the changing precautions they were taking, their changing comfort with social gatherings, their use of stimulus payments and the state of their finances.

Spending in shops was convulsing, literally down 17.7% one month (on lockdowns), then up 17% the next (on panic buying).

Delays unacceptable

Yet the retail figures had always been presented with a delay — four to five weeks after the month to which they referred — while the bureau waited for all of the retailers it was surveying to report, making the insights anything but current.

Gruen got the bureau to release “preliminary” numbers two or three weeks earlier than usual, as soon as 80% of the businesses surveyed had responded.

Information about deaths (rather useful in a health crisis) was even worse.

Not information about COVID deaths, which heath authorities were totting up daily, but deaths from all causes, which the bureau traditionally released once a year once all the reports from coroners had come in, every September, almost an entire year after the year in which the deaths took place. Some of the deaths were the best part of two years old.

Provisional now, final later

Gruen suggested that rather than wait until every coroner’s report was finalised, the bureau release “provisional mortality statistics” based on only doctor-certified deaths (80-85% of all deaths) monthly.

What it showed was startling. Rather than having more deaths than normal from non-COVID causes, as had much of the world, Australia had fewer.


Read more: Up to 204,691 extra deaths in the US so far in this pandemic year


The excess deaths in other countries might have been either COVID deaths not classified as COVID deaths, or deaths inadvertently caused by measures designed to fight COVID.

In net terms Australia has had neither. The bureau’s figures show we’ve been less likely than normal to die of heart disease and strokes, and far less likely to die from flu, probably because social distancing has made it harder to catch.


ABS Provisional Mortality Statistics

As incredibly useful as these innovations have been, none has been as valuable as the bureau’s inspired decision to obtain and publish near real-time payroll data.

What took months is now near-instant

The Tax Office is phasing in a requirement for businesses to report where they send their payroll instantly using a system it calls single-touch payroll. 99% of big and medium firms (20 or more employees) are doing it, and 75% of small firms.

It is data on 10 million jobs updated weekly, broken down by gender, age, industry and location — near-instant data of the kind Australia has never seen.

The treasury has been able to use it to fine tune (and change) its programs as the crisis was unfolding.

Detail like never before

And there’s more. The bureau is going to use single-touch payroll to come up with a near-instant monthly measure of earnings. It is going to use business activity statements provide an near-instant measure of business turnover.

And it has got the big four banks to hand over aggregated consumer spending data far more comprehensive than the subset that finds its way into the retail trade survey.

It is also experimenting with using anonymised electricity smart meter data to work out the extent to which people are staying at home, and using deidentified data from mobile devices to work out the extent to which we are moving about.


Read more: GDP is V-shaped, but not yet good. These three graphs tell the story


If the government has got most things right in the economic management of the crisis, it is largely because it has known more about the granular detail of what’s been happening than any government before it.

With one forecast suggesting hundred of thousands of Australians could lose their jobs when JobKeeper ends on March 28, and the impact of the extra measures the government will unveil this week uncertain, it’ll keep needing to know.

Australian Statistician David Gruen at Economic Society of Australia annual dinner Canberra Octoer 28 2020

Peter Martin, Visiting Fellow, Crawford School of Public Policy, Australian National University

This article is republished from The Conversation under a Creative Commons license. Read the original article.

Read more >>

Thursday, August 17, 2017

Same-sex marriage.The one good thing about the survey

The ABS is fighting back.

Forced to conduct a survey that its staff know will almost certainly be unreliable and possibly wrong, it has planted a time bomb.

First, some history. The Bureau of Statistics is the best organisation in the country at getting at the truth, whatever the statistical question.

It does it in census by taking the extraordinary step of surveying the entire population, and making answers compulsory.

It does it in surveys like the employment survey by sampling a portion of the population (26,000 households) and multiplying the results by around 300 to develop a picture of the entire nation.

But it doesn't simply multiply by 300. The sample will be biased. Some groups will be under-represented, such as women in their 60s. Where that happens, the Bureau gives the responses of women in that age group extra weight to compensate for the bias. If there are half as many women aged in their sixties as you would expect from the Australian population, the Bureau counts their responses twice to make the survey representative. If there is twice as many 20-something men as you would expect, the Bureau averages their responses and applies them to half the number of men.

It's important. In a fortnight the Bureau will release the results of its long-awaited six-yearly survey of household expenditure. It will update our views about how much we spend on technology, how much we spend on electricity and so on. It's a survey of 9800 households. But unless those households contain exactly the right mix of teenagers, pensioners and every other type of Australian they won't tell us much or be comparable to previous surveys unless they are adjusted to properly represent the population.

That's what the Bureau usually does, and what it did the last time it was asked to conduct a plebiscite-like survey, in 1974. It surveyed 60,000 Australians to in order to find out what national anthem we preferred, and – just as with the employment survey – it weighted the results. The man who ran the poll, the then assistant statistician Bill McLennan, says anything else – anything uncorrected for bias – would have been "rubbish".

I've seen McLennan's printout, broken down by age and gender. You can tell how men aged 18 to 24 voted, and how women aged 65 to 69 voted. And you can see the totals after they have been adjusted to remove bias and accurately represent the entire Australian population.

This time there are exceptionally good reasons for removing bias. Malcolm Turnbull expects roughly half of those eligible to respond. But it won't be just any half. History suggests it'll be predominantly older Australians. ABC election analyst Antony Green says the last time we conducted a voluntary plebiscite, in 1997 to select delegates to the Constitutional Convention, younger Australians were almost half as likely to take part as older ones.

We know from the privately-conducted surveys that younger people are far more likely to support same-sex equality an older Australians.

And this time they may be even more poorly represented than in 1997. Many don't know how to post letters, many more don't know where to find them.

Best practice as normally applied by the ABS, even in the hugely representative census, is to give a greater weight to the under-represented responses and a lesser weight to the over-represented ones.

But the Bureau can't do it. It's been ordered not to. The ballot has to be secret, which makes it impossible. It will open envelopes that will have all the information needed to identify each type of respondent, and then separate them from the responses, which will be pooled electorate by electorate, making them indistinguishable.

Australian National University demographer Liz Allen says of all the methods available to work out what Australians really think about same-sex marriage, a voluntary postal survey is the least likely to produce the right result.

Even if the Bureau knows the responses are chronically biased, there's nothing it can do, except wish it had resisted.

McLennan believes it could have. He acknowledges that the Census and Statistics Act allows the treasurer to direct the statistician to collect statistics on a specific topic, but he says after that it's up to the statistician to decide how to do it and when and how to publish the results.

Mr Morrison has directed the statistician to collect "statistical information from all Australians on the electoral roll as to their views". McLennan believes best way to ascertain the views of all Australians on the roll is to survey them and weight the results to represent the roll.

Or it can passively resist. Buried within the Bureau's rundown of how it will go about the poll is its decision to publish response rates by electorate, gender and age. On November 15 when the results are out it'll be instantly apparent whether the poll is representative; whether older Australians, or women or men or West Australians are over or under represented. If they are, if the results are "rubbish", the Bureau will have made sure we know.

In The Age and Sydney Morning Herald
Read more >>

Thursday, October 27, 2016

I've got how long? Life expectancy hits new high

Life expectancy has hit a new high, with typical newborn girls now expected to live to 84.5 and boys to 80.4, up from 83.3 and 78.5 a decade ago.

New life tables from the Bureau of Statistics show a typical 30-year-old woman can expect another 55 years, with a further 36 years for a 50-year-old, 18 for a 70-year-old, and 2.4 for a typical 100-year-old.

For men, a typical 30-year-old will get another 51 years, with 32 years for a 50-year-old, 15.6 for a 70-year-old, and 2.2 for a typical 100-year-old.

Men at the traditional retirement age of 65 have another 19.5 years. Fifty years ago in 1966, they had only 12 years. Women at 65 have another 22.3 years. Fifty years ago they had 15.7.

The Australian Capital Territory has the highest life expectancy of 85.3 for newborn girls and 81.2 for boys, followed by Victoria at 84.7 and 81.1. The Northern Territory has the lowest, of 78.5 for girls and 75.7 for boys. Tasmania is the second lowest at 82.8 and 78.9. 

Indigenous life expectancies at birth are 9.5 years lower for girls and  10.6 years lower for boys.

But the reality is less grim. Academic demographers believe the ABS figures are almost certainly underestimates.

Peter McDonald, of the University of Melbourne, says they assume no improvements over the course of a life.

"They are not any individual's lifetime; they are just telling you the expectation of life you would get if life expectancy didn't change," he said. "And for the last 200 years it has been going up."

The ABS itself says the figures it quotes for life expectancies at birth are the average number of years that a group of newborn babies would be expected to live "if current death rates remain unchanged".

Professor McDonald said a rule of thumb was that improvements over an 85 or 81 year life would add anther four  years. That means a typical newborn girl might live to 88 and a typical boy to 85, unless improvements stop or accelerate.

In The Age and Sydney Morning Herald
Read more >>

Wednesday, October 26, 2016

Census debacle: Turnbull to decide which heads to roll

The Prime Minister's special adviser on cyber security has told the Senate the denial of service attacks on the census website were small and predictable and should not have brought it down on census night.

Malcolm Turnbull now has the report Alastair MacGibbon conducted on behalf of the Prime Minister to determine "which heads will roll and when" as a result of the debacle.

"They were indeed small attacks," Mr MacGibbon told a Senate committee on Tuesday. "The attacks were around three gigabits per second. To have some comparison, it's not uncommon now to see attacks of 100 gigabits per second, and some of the attacks against some of the internet infrastructure such as domain name servers are up to 1000 gigabits per second.

"There was a massive difference between the size of the attacks on the Bureau of Statistics' census website and the ones that are encountered routinely by corporations and governments."

While the bureau had contracted IBM to defend its sites against attacks, its behaviour after awarding the contract was similar to that of a homeowner who employed a builder but then rarely went on site to check how work was progressing, he said.

The bureau's back-up plan to protect the site if denial of service attacks couldn't be overcome was logically flawed.

Labelled "Island Australia", it was to ask IBM to block traffic from overseas. But the password reset facility IBM used was hosted offshore and relied on traffic coming in from overseas to give Australians that password, suggesting it hadn't been properly thought through.

Larger failures were that IBM was unable to implement Island Australia in any event and that ABS staff misread a report they thought suggested census data could have been leaving the system as a result of hacking and decided to shut the system down.

IBM was for many hours unable to restart it because it had incorrectly coded a router connecting to Telstra, so that when it was turned off the coding "fell out", turning it into a "dumb unit" that had to be recoded.

Had the router been turned off and then turned on again as a test, the error would have been discovered.

"Had the router been properly configured, and had the router when it had been turned off fired back up again, then we wouldn't have a problem," Mr MacGibbon said. "But the most significant problem was really the misinterpretation of the traffic on the load monitoring system. We wouldn't have had the problem if the people monitoring the system had properly monitored the system, which was functioning oddly."

Millions of Australians were unable to complete the census on census night as a result of the shutdown and were locked out of the site for two days.

Mr MacGibbon delivered his report to Mr Turnbull on October 14.

IBM Australia managing director Kerry Purcell told the hearing no IBM staff had been dismissed as a result of the failure of the census website and none had been disciplined.

IBM had offered to pay the extra costs the ABS incurred as a result of the outage, estimated by the ABS to be $30 million. It is in "commercial negotiations" with Secretary of the Treasury John Fraser.

Mr MacGibbon also criticised the closeness of the bureau to IBM, saying there was a degree of "vendor lock-in", where the ABS saw IBM as a natural partner because it had worked with it in the past.

A representative of Capability Driven Acquisition, the company that advised the ABS on hiring IBM, said several other potential bidders had told it there was little point in competing against IBM because it would win the contract.

The bureau's chief, David Kalisch, told the committee he would have considered an open tender had "IBM not been able to satisfy the ABS that it could deliver".

One of many "learnings" the bureau had taken from the experience was that it might be worthwhile running the next census in-house and that the slogan "Get Online on August 9" may have contributed to the problem.

Mr Kalisch defended the bureau's decision to retain the names submitted with this year's census and revealed that in the past no one who declined to submit their name had ever been prosecuted.

A former head of the bureau, Bill McLennan, told the the hearing that in his time the bureau had received legal advice telling it that it lacked the power to compel people to provide names.

In The Age and Sydney Morning Herald
Read more >>

Thursday, October 20, 2016

ABS chief: census meltdown cost $30 million and I'm sorry

The head of the Bureau of Statistics has apologised for poor judgment and testing the patience of Australians during the 2016 census, and revealed that the failure of the digital system cost taxpayers $30 million. 

Reading from a prepared statement at the start of a Senate estimates hearing, David Kalisch said he wanted to thank the Australian population for "their forbearance and diligence" in completing the census which was taken offline at what would have been the peak period for submitting forms.

"The ABS tested the patience and commitment of many households especially through the difficulties accessing the call centre and the unavailability of the census online form for nearly two days," he said.

"We made a difficult decision to take the system off line on August 9 to ensure the security of census data, but we should not have got to that point, and the system should have been robust to denial of service events."

"The ABS made a number of poor judgments in our preparation for the 2016 census that led to the poor service experienced by many households. I apologise to the community on behalf of the ABS and I repeat that apology sincerely again today."

Mr Kalisch revealed that while the move to a predominantly digital rather than paper-based census had been intended to save $100 million, the attempts to recover from the system shutdown cost another $30 million, cutting the net saving to $70 million.

"We have to date probably incurred additional costs of around $20 million ... and we anticipate possibly spending another $10 million," Mr Kalisch said.

The census website suffered a further denial of service attack after it had been put back on line, one the Bureau and its contractor IBM put a lot of effort into successfully repelling.

The Bureau also badly handled its bid to retain the names submitted with the census, which had previously been disposed of after processing.

It allowed only three weeks for comment and received only three submissions.

"I think it's probably fair to say there should have been a longer consultation process," he said. "We should have planned that better."

Ninety six per cent of households appear to have completed the census, more than the target of 93.3 per cent, but a full accounting won't be possible until March or April next year.

Fifty eight per cent completed the forms on line, much less than the target of 65 per cent.

Prime Minister Malcolm Turnbull has ordered an inquiry into the census to determine "which heads will roll and when".

Mr Kalisch was appointed to the $705,030 post in  2014. He will appear before a specially-convened Senate inquiry into the census next week.

In The Age and Sydney Morning Herald
Read more >>

Thursday, October 13, 2016

Driving without lights: ABS prepares to cut critical surveys

A decade ago in the lead-up to the global financial crisis, the Bureau of Statistics dimmed the lights. It suspended its job vacancies survey and slashed its employment survey by a quarter.

When the crisis hit, the new treasurer, Wayne Swan, was in the dark. Without the job vacancies data, he didn't have a check on employment data that had itself become erratic and unreliable at the local level. He and the Reserve Bank had little idea how things were panning out across the country.

The Bureau eventually got extra funds and restored the surveys (after the worst of the crisis had passed), but more recently it has been cut and cut again.

Labor cut $10 million on the way out in 2013 and the Coalition cut $68 million on the way in in 2014. So hard up was it before it got a reprieve in the lead-up to this year's census that it asked the government for permission to abandon it so it could use the funds to fix its antiquated computer systems. In 10 years its workforce has shrunk by one sixth.

On Thursday, its chief David Kalisch revealed it did "not have the resources to undertake all the activities that fall within our legislative mandate that our users would like".

This time it is leaving the job vacancies survey alone. Instead it's going after the monthly retail sales survey as well as those on housing and lending finance, and a number on international trade. It says it will consult with users about making them less frequent, which will make them less useful.

The Reserve Bank in particular relies on the monthly housing finance survey to give it a handle on what happens as soon as it moves interest rates. It's what CommSec chief economist Craig James calls "forward looking" data. Applications for home loans provide a good guide as to what's about to happen to home building.

HSBC economist Paul Bloxham says retail sales and borrowing figures are important for more than just the Bank. Individual companies and retailers themselves are economic managers. They need to know what's going on.

Deloitte Access Economics director Chris Richardson says we'll be able to live with the cuts "until we can't".

"Until the next financial crisis or whatever it is rolls through town and we're desperately trying to get a pulse, or indeed find one, that's when we'll know what's missing," he says.

Many more surveys are up for axing altogether, unless user funding is secured, among them those on internet use, the experience of patients in hospitals, victims of crime, and the foreign ownership of agricultural businesses.

Labor's Andrew Leigh quipped that given the government's success with the national broadband network, it probably didn't want surveys about internet speeds, and given its war on Medicare it probably don't want surveys about patient experience.

The cuts came on top of "what had to be a contender for the worst managed census anywhere in the world in 2300 years".

It might be a cynical bid for more funds, as some say the earlier cuts were. If so, we should consider handing them over. We need the surveys.

In The Age and Sydney Morning Herald
Read more >>

Friday, September 23, 2016

Blame all around as the ABS spreads responsibility for census

The Australian Bureau of Statistics has blamed the media for the failure of its census hotline and blamed an overseas denial-of-service attack for the failure of its census website.

In a strongly worded submission to a Senate inquiry, the Bureau also attempts to deflect blame for the overwhelming of its website on to its contractor, IBM.

The submission says from early on, both its call centre and its automated paper-form request service received far more calls than expected, forcing it to implement a "call blocking" where calls were answered and callers asked to call back later.

It blames "unexpected and unprompted media and social media focus on potential of census fines", for which it hadn't planned because its usual strategy is not to mention fines before the census night. Also Australia Post delivered letters informing people of the hotline more quickly than it expected and its census advertising campaign was more effective than it expected.

In media reporting about its decision to retain names submitted with the census to enable linking to other datasets, the ABS was "rarely approached for its perspective". It met with the Australian Privacy Foundation to explain its position but the Privacy Foundation "continued to reflect their views irrespective of ABS explanations".

Before the denial-of-service attacks, which caused it to shut down the website on census night, its contractor, IBM, had provided "reasonable assurances" that adequate protections were in place.

"At no time was the ABS offered or advised of additional protections that could be put into place. Additionally, no suggestion was made to the ABS that the protections that were planned were inadequate," the submission says.

After the first denial-of-service attack at 10.10am on census day, the ABS asked IBM to invoke "Island Australia", where the website was cut off from other countries.

By the fourth, at 7.28pm, the ABS and IBM observed "an unusual spike in outbound traffic". The ABS instructed IBM to prevent the Australian public from commencing new census forms to ensure the census data was protected.

It was unable to shut down web advertising asking people to file online or automated Twitter responses telling people to keep trying for some time.

As of September 20, 94.4 per cent of households had completed their census return, in line with expectations. Fifty-nine per cent had completed their forms online, less than expected.

By September 20, 6743 people had refused. During the previous, 2011, census 13,194 people refused.

In The Age and Sydney Morning Herald
Read more >>

Saturday, August 13, 2016

Code Red: How the ABS bungled the census

In 2015, at what ought to have been the height of preparations for the 2016 census, its head, Duncan Young, sent his colleagues in the Bureau of Statistics executive a crisis memo.

"As most of you are aware, the census program has alerted its steering committee and board that it has assessed its status as RED," he wrote in February.

He put the word "RED" in capital letters.

"This means that we have assessed that the program will not be able to deliver on the current scope, timetable and/or budget. This status is as a consequence of both budget reductions since program commencement and program delays during 2014."

Young set out five options to shrink the census or save money, each "not taken lightly". One was to ask far fewer questions, another was to reach fewer people.

At the same time the Bureau's newly installed chief, David Kalisch, was war-gaming an even grander solution. Dubbed "Project Archer" after Keith Archer, the Australian Statistician who introduced computers to the ABS in the 1960s, it would make the 2016 census go away, freeing up $200 million to $400 million to upgrade the Bureau's aging computer systems, some of which ran code that was 30 years old.

His predecessor Brian Pink had left at the start of 2014, warning in his final annual report that the Bureau had barely enough cash to "keep the lights on".

When Pink arrived in 2007 the Bureau received $302 million in a non-census year. Seven years of growing expenses and relentless "efficiency dividends" later, it received scarcely any more, $312 million.

Pink had responded by axing or indefinitely postponing some of Australia's most loved surveys. One was the national time-use survey which records how Australians spend every 15 minutes. It hasn't been updated since 2006, before the arrival of the iPhone.

Kalisch was interviewed for the $705,030 job at the start of 2014 but wasn't appointed until December, leaving the Bureau without a leader during a year in which it was meant to be fine-tuning Australia's first predominantly digital census.

Tony Abbott and his treasurer Joe Hockey dithered because they didn't like the look of the candidate the selection committee had first recommended. Project Archer would deliver the census once every 10 years instead of five, directing the savings to buy new computers.

To sell the idea, Kalisch had to diss the census...

"There is a lot of, perhaps, misinformation about the value of census," he told a Senate hearing. "There is a sense in the community that a lot of the information is derived from the census, which is just not true."

The primary purpose of the census is indeed prosaic – it is to count the number of people aged 18 and over in order to determine the shape of electorates. It's why politicians are particularly keen on keeping it, and why cancelling it was always a big ask. The government said no, and gave the Bureau an extra $235 million over five years in order to upgrade its computer system.

But the census itself had to be cut-price, costing more like $200 million than the previous $300 million to $400 million. Work on the questions as good as stopped.

Every five years there's an additional special-interest question. It's incredibly valuable real estate, fought over in the same way as the payload on a mission to space.

In 2001 as Peter Costello was gearing up to replace John Howard as Australia's prime minister, he gave a speech about the "spirit of the volunteer" in an attempt to humanise himself.

As treasurer, he instructed the ABS to make the 2006 special question about volunteering, even though it already collected more detailed statistics on volunteering in another survey.

The Bureau reluctantly complied, and then when the Rudd government left it short of funds in the lead up to the 2011 census, left it in because it didn't have the money to devote to framing another question. By the lead up to this census it was short of money again and desperately short of time. It left in the ill-defined question for the third consecutive census.

Directed to actually conduct the census, and keen to extract some value from it, Kalisch and his team revived an idea categorically ruled out by his predecessor. Pink had said no to retaining names.

"It wasn't going to happen. I can tell you that," Pink said this week.

"I always used to say to my people: you can't kill the goose that lays the golden egg, and the golden egg is the census. In my view, you only need 20 per cent of Australians who are concerned about security and you put the census at risk."

When given the option of having their names and forms retained and stored in the archives for release a century later instead of being destroyed after processing, 39 per cent of Australians had said no. The immediate use of their names might have alarmed them more.

Names had always been retained for a short time in order to eliminate duplicates and establish the relationship between household members, but destroyed after checks, usually well before 18 months.

On October 19, Kalisch convened a meeting of his executive group. It agreed to conduct a privacy impact assessment into the permanent retention of names as well as exact addresses, which had also previously been destroyed after checking.

Whereas in Pink's day the privacy impact assessment had been conducted externally, and had savaged the proposal, this one would be conducted in-house "consistent with our practice with data integration projects and leveraging the experience and knowledge we have built since 2005".

Its publication along with a half-hearted endorsement from focus groups conducted by Colmar Brunton Social Research would be timed "to quickly follow" the release of the regular Trust in ABS survey which always produced impressive results.

Appearing before the Senate economics committee two days later, Kalisch said nothing about the plan to retain names and addresses. He made a short statement to "update the committee about our census preparations". Things were "on track" and momentum was building. A few months earlier he he had told the committee things were coming along "beautifully".

On December 8 the executive group considered the proposal in more detail. The ABS had published a statement of intent on its website on November 17, unreported in the mainstream press, and received just three responses, all negative, from what it termed "concerned private citizens".

The internal privacy impact assessment had given it a tick. The Bureau had sent a minute to the office of the assistant minister to the treasurer Alex Hawke, appointed a few weeks earlier by the new Prime Minister Malcolm Turnbull. A report prepared for the meeting said it had been "noted".

The Bureau wanted to build a reputation as Australia's "premier integrator of government data". If it couldn't retain names and addresses, potential users might see it as "unnecessarily constraining itself and therefore constraining whole-of-government data integration".

"There are many administrative datasets that are likely to have considerable statistical value," the report said. "In addition to the personal income tax data which has already been used in data integration projects, future data integration projects could include the use of welfare payments data, Centrelink unemployment benefits data, Medicare and Pharmaceutical Benefits Scheme data, Australian Immunisation Register, the electoral roll, and other nationally important datasets."

The report envisioned no limit on what the ABS could link and charge for, so long as the names and addresses themselves were kept within the ABS. Information from the census on ethnic or religious backgrounds could be linked to information from the immunisation register to work out what type of families on what types of incomes were the least likely to immunise.

Criminal records could be linked to census records, if permission were given, to see what sort of Australians were convicted of what sort of crimes.

Until that point the Bureau had mainly relied on "bronze" linkage – the rough linking of files using identifiers other than names and addresses. A move to "gold" linkage using names and addresses would get "maximum value for what is already one of the most valuable statistical assets the ABS holds".

On the Friday before Christmas, the Bureau released an eight-paragraph statement deceptively titled ABS response to Privacy Impact Assessment. Once more unreported in the mainstream media, it said the Bureau would retain the names and addresses collected in the census "to provide a richer and dynamic statistical picture of Australia through the combination of census data with other survey and administrative data".

In April, with the census imminent, after reports in Crikey and the Australian Financial Review, Kalisch backed down somewhat. Names would be kept for only four years, but the really useful linkage keys derived from them would still be kept indefinitely.

One of Kalisch's predecessors, Bill McLennan described what was planned as "without doubt, the most significant invasion of privacy ever perpetrated on Australians by the ABS".

"I am appalled that the ABS can think it can use the threat of prosecution to make me provide data that allows the ABS to set up what is, in effect, a statistical Australian Card," he wrote.

As it moved to counter declarations by Crikey reporter Bernard Keane and high-profile politicians including Nick Xenophon that they would either not complete the census or not provide their names, the Bureau emphasised the $180 per day fines. They applied for each day the forms weren't complete, without limit. Although the Bureau was also careful to point out that they applied only after September 23, the main message received was that the forms had to be completed on census night itself, August 9.

By 7.30pm, as millions of Australians tried to get online at once amid what may have been denial of service attacks, the system crashed and was taken down. It had been built by IBM for $9.6 million and load-tested by Revolution IT for $469,000. ABS robots, set up to automatically respond to tweets, encouraged Australians to continue to try to log on.

Kalisch had said just the day before the Bureau was "ready" with the best security features for which "you could ever ask".

The new minister, Michael McCormack, in the job for mere weeks, at first couldn't get through to Kalisch. McCormack had been appointed after an embarrassing interlude in which there seemed to be no minister responsible. Hawke's position had been abolished and neither treasurer Scott Morrison nor financial services minister Kelly O'Dwyer had been given the job. A fortnight after being appointed small business minister McCormack was told it was his.

Before the website went back online late Thursday Prime Minister Malcolm Turnbull promised an inquiry after which he said "heads would roll".

In The Age and Sydney Morning Herald
Read more >>

Wednesday, August 10, 2016

Census meltdown just the latest Bureau of Statistics bungle

"Reckless" doesn't begin to describe the new culture at the top of the Bureau of Statistics.

Its most important product apart from the census is the monthly employment survey. Two years ago it decided to modernise. It moved much of it online, accepted a lower response rate and changed the months in which different questions were asked.

Against the advice of ABS veterans, it didn't run a backup survey using the old system.

In August 2014 the number of Australians officially employed jumped an incredible 121,000. The next month, September, it dived 172,000, or it would have had the ABS not pleaded with users to ignore the seasonally adjusted numbers which it no longer trusted.

Without a backup – a parallel employment survey conducted under the old system – it was impossible to tell what was wrong and what was right....

With no leader, treasurer Joe Hockey and prime minister Tony Abbott had left the position at the top unfilled for the best part of a year, the ABS developed grander plans.

It wanted to abandon the 2016 census altogether, moving from five-yearly to 10-yearly, to save $200 million.

Told by the government that would require legislation that would be unlikely to get through the Senate, it decided to get extra value by using the names collected with the census to create permanent linkage keys. Previously destroyed after processing, the names would be kept indefinitely in a separate file and used to link the answers to census questions to the answers to other ABS surveys and data collected from organisations such as the Pharmaceutical Benefits Scheme and the Australian Tax Office. If it could find out whether Australians or certain ethnic backgrounds or family circumstances were more likely to claim certain tax deductions or be prescribed certain medicines it could sell the results.

The only problem was that it had put up the idea before to an independent privacy impact assessment, which had savaged it. So it conducted its own assessment in house, which found no problem. It published the results quietly late last year (there was nothing about in in the mainstream press) and then on the Friday before Christmas announced it would retain every name indefinitely in a press release with the misleadingly bland title of: ABS response to Privacy Impact Assessment.

By April it had backed down. It was going to keep the names for only four years, but it would keep the linkage keys created from them for as long as would be needed, which could be forever.

Woefully unprepared to explain why it now wanted to retain names, it emphasised instead the digital nature of the census. It would save millions by posting login codes to most of the population rather than delivering forms. Had it delivered, or even posted, forms it would have had a backup.

Instead it gave most of Australia only one way to submit census forms, emphasised the importance of the survey it had previously tried to ditch, threatened fines of $180 per day for people who didn't comply, and underestimated either the strain on the system or the security of the system.

At almost every step of the way the government has been hands off. The latest minister (Michael McCormack has been in charge of the ABS for less than two weeks) gives the impression the decision to retain names didn't even go to cabinet.

There's already been an inquiry into the ABS. It found it was not properly ready "to maximise the value of all government-held information". Which might be part of the problem. Until now one of Australia's most trusted organisations, it has tried to catch up too fast. The government has looked the other way.

In The Age and Sydney Morning Herald
Read more >>