Wednesday, January 09, 2013

The shocking truth about forecasting

Wednesday column

Whoops. As far as forecasting errors go, the latest confessed to by the International Monetary Fund is a whopper. It’s in a category of its own.

Other contenders might be the man who turned down the Beatles (“groups with guitars are on their way out”) or the movie executive who said home video recorders would kill the motion picture industry (“I say to you that the VCR is to the American film producer and the American public as the Boston strangler is to the woman home alone”).

But those misforecasts never caused massive human misery (all the more so because they weren’t taken seriously).

The IMF on the other hand solemnly advised the nations of Europe coming out of the financial crisis to raise taxes and wind back government spending. Its commandments had weight. Yes, it had failed to foresee the crisis in the first place, but it was the lender of last resort. They might need it.

And it had modelled what would happen if they did what it said. For every dollar they cut their budgets their economic growth would suffer just 50 cents. Its forecasts said so.

On Friday in its first working paper of the year it revealed the full horror of what did happen. Personally authored by the Fund’s chief economist Olivier Blanchard the mea culpa says for every dollar those nations cut their budgets their economies crumpled something more like 150 cents.

That’s right. Rather than suffering far less than the savings they made on their budgets, their economies suffered far more. As mistaken advice it’s monstrous - like going to see a doctor who tells you the medicine won’t hurt much and finding it lays you low for years.

The Fund forecast that if the Eurozone took its advice it would grow 1.8 per cent throughout 2011. It grew 0.7 per cent. Italy would climb 1.3 per cent, it slid 0.5 per cent. Spain would surge 1.8 per cent, it grew not at all.

The errors are completely unlike those made forecasting the Australian budget which were largely the result of unexpected events.

The shocking thing about the IMF misforecasts confirmed by its chief economist is that there were few unexpected events... The global environment was broadly as forecast. The nations of Europe did what was forecast. The consequence was nothing like what was forecast. The Fund misunderstood the mechanics.

As Olivier Blanchard put it, his forecasters “significantly underestimated the increase in unemployment and the decline in private consumption and investment associated with fiscal consolidation”.

His defence is that in normal times they would have got it right. In normal times a budget cut of one dollar would have cut economic growth by 50 cents. But the times weren’t normal. European interest rates had been cut to nearly zero, meaning there wasn't the normal room for authorities to cut further, and households were more heavily indebted than normal meaning cuts to their income flowed through more quickly than normal to cuts in their spending. And the starting point was different. The European economies had been in recession, which was far from normal.

You or I might say the forecasters didn’t do a thorough job. The Wall Street Journal says they may have been ‘intellectually lazy’.

But they weren’t alone. The European Commission, the Organisation for Economic Co-operation and Development and the Economist Intelligence Unit all made the same mistake. The IMF tells us so.

When forecasters are wrong, they are usually wrong together.

A year ago only two of the panel of 20 professional forecasters assembled for this newspaper’s BusinessDay economic survey picked Australian dollar above $US1. They were the only two that were right. This time only four picked an Aussie below $US1.

There’s a reward for staying with the pack. You’ll keep your job if you are wrong in good company, even if the people who act on your advice lose the lot.

The shocking and little-acknowledged truth is that most expert forecasts are wrong. Not only wrong, but more wrong than if they had been generated at random.

Two decades ago psychologist Philip Tetlock of the University of California Berkeley began testing the forecasts of 284 famous Americans who made their living pontificating about politics and economics. As he says in his book Expert Political Judgement: How Good Is It? it wasn’t easy to pin them down. When stripped of rhetoric their predictions were surprisingly slippery.

So he surveyed them asking every few months whether the variable they covered would (a) stay the same, (b) increase or (c) decrease.

More than 82,000 testable forecasts later he found that as a group the experts performed worse than if they had just selected (a), (b) then (c) in rotation. They performed worse than a dartboard.

Two Reserve Bank economists have just found the same thing about the Reserve Bank itself. One year out its unemployment forecasts have been “less accurate than a random walk”.

There are exceptions. Weather forecasters are especially good, as we are discovering right now. The New York Times data geek Nate Silver got the presidential election spot on. These exceptions tell us something. Neither Silver nor our weather forecasters think they are experts (Silver comes from sports rather than politics). They are guided by the data - regardless of who it offends - rather than their own judgement.

By contrast experts have reputations to protect. Whether they realise it or not they often play games, avoiding intellectual curiosity if it will leave them out on a limb away from the pack. They remember their good forecasts and bury the bad. Put plainly they are not the sort of people you would want providing economic advice that could have catastrophic consequences.

Nassim Nicholas Taleb, author of The Black Swan: The Impact of the Highly Improbable and the new book Antifragile asks why predictors keep predicting, given that their predictions are so often wrong. His answer: “They are not harmed by what they are doing”.

He says instead of asking a doctor what you should do, you should ask what the doctor would do him or herself.

In today's Sydney Morning Herald and Age


Recommended listening

. Antifragile: how to live in a world we don't understand - LSE talk by Nassim Taleb

Related Posts

. Who says most Reserve Bank forecasts are hopeless? Err...

. Good at picking stocks. Not.

. Are Some Forecasters Really Better Than Others? Oh my

. Conformity. The candid camera experiment is only the beginning