Thursday, August 17, 2017

Same-sex marriage.The one good thing about the survey

The ABS is fighting back.

Forced to conduct a survey that its staff know will almost certainly be unreliable and possibly wrong, it has planted a time bomb.

First, some history. The Bureau of Statistics is the best organisation in the country at getting at the truth, whatever the statistical question.

It does it in census by taking the extraordinary step of surveying the entire population, and making answers compulsory.

It does it in surveys like the employment survey by sampling a portion of the population (26,000 households) and multiplying the results by around 300 to develop a picture of the entire nation.

But it doesn't simply multiply by 300. The sample will be biased. Some groups will be under-represented, such as women in their 60s. Where that happens, the Bureau gives the responses of women in that age group extra weight to compensate for the bias. If there are half as many women aged in their sixties as you would expect from the Australian population, the Bureau counts their responses twice to make the survey representative. If there is twice as many 20-something men as you would expect, the Bureau averages their responses and applies them to half the number of men.

It's important. In a fortnight the Bureau will release the results of its long-awaited six-yearly survey of household expenditure. It will update our views about how much we spend on technology, how much we spend on electricity and so on. It's a survey of 9800 households. But unless those households contain exactly the right mix of teenagers, pensioners and every other type of Australian they won't tell us much or be comparable to previous surveys unless they are adjusted to properly represent the population.

That's what the Bureau usually does, and what it did the last time it was asked to conduct a plebiscite-like survey, in 1974. It surveyed 60,000 Australians to in order to find out what national anthem we preferred, and – just as with the employment survey – it weighted the results. The man who ran the poll, the then assistant statistician Bill McLennan, says anything else – anything uncorrected for bias – would have been "rubbish".

I've seen McLennan's printout, broken down by age and gender. You can tell how men aged 18 to 24 voted, and how women aged 65 to 69 voted. And you can see the totals after they have been adjusted to remove bias and accurately represent the entire Australian population.

This time there are exceptionally good reasons for removing bias. Malcolm Turnbull expects roughly half of those eligible to respond. But it won't be just any half. History suggests it'll be predominantly older Australians. ABC election analyst Antony Green says the last time we conducted a voluntary plebiscite, in 1997 to select delegates to the Constitutional Convention, younger Australians were almost half as likely to take part as older ones.

We know from the privately-conducted surveys that younger people are far more likely to support same-sex equality an older Australians.

And this time they may be even more poorly represented than in 1997. Many don't know how to post letters, many more don't know where to find them.

Best practice as normally applied by the ABS, even in the hugely representative census, is to give a greater weight to the under-represented responses and a lesser weight to the over-represented ones.

But the Bureau can't do it. It's been ordered not to. The ballot has to be secret, which makes it impossible. It will open envelopes that will have all the information needed to identify each type of respondent, and then separate them from the responses, which will be pooled electorate by electorate, making them indistinguishable.

Australian National University demographer Liz Allen says of all the methods available to work out what Australians really think about same-sex marriage, a voluntary postal survey is the least likely to produce the right result.

Even if the Bureau knows the responses are chronically biased, there's nothing it can do, except wish it had resisted.

McLennan believes it could have. He acknowledges that the Census and Statistics Act allows the treasurer to direct the statistician to collect statistics on a specific topic, but he says after that it's up to the statistician to decide how to do it and when and how to publish the results.

Mr Morrison has directed the statistician to collect "statistical information from all Australians on the electoral roll as to their views". McLennan believes best way to ascertain the views of all Australians on the roll is to survey them and weight the results to represent the roll.

Or it can passively resist. Buried within the Bureau's rundown of how it will go about the poll is its decision to publish response rates by electorate, gender and age. On November 15 when the results are out it'll be instantly apparent whether the poll is representative; whether older Australians, or women or men or West Australians are over or under represented. If they are, if the results are "rubbish", the Bureau will have made sure we know.

In The Age and Sydney Morning Herald