Who predicts well?

…what separated those with modest but significant predictive ability from the utterly hopeless was their style of thinking. Experts who had one big idea they were certain would reveal what was to come were handily beaten by those who used diverse information and analytical models, were comfortable with complexity and uncertainty and kept their confidence in check.

That is from Tetlock and Gardner, here is more.  On that general, theme, here is Dan Gardner’s new book Future Babble: Why Expert Predictions are Next to Worthless, and You Can do Better.


Tetlock's experiment in crowd-casting intrigued me. He asks for volunteers. Unfortunately, he only accepts those with degrees. High school dropouts like myself need not predict. Won't that bias his results?

Hmmm, traders make predictions about the future all the time, and they're probably not as far off as all of these "experts". I don't think that's because they have more of a grasp of the complexity of the outcomes they're predicting; I bet it's because they have an objective scorecard that shows whether they've made good or bad predictions (i.e., the money they've earned on a trade).

I bet if these other experts had a similar scorecard they'd quit being so reckless with their words

Traders probably hit the mark more often than experts because most traders aren't experts. In fact, most traders are basically excitably frat boys playing an aggressive game only until they make their first big haul.

There is always someone on the other side of the trade. So looking very closely at the 'winners' in trading, you often find a selection bias. Taleb has made a career of showing that traders have a bias toward low risk-low reward trades which allows them to show good results over time: until they blow up. They blow up because the low risk, is only apparent.

The old fashioned "value investing" approach tends to work because of this loss aversion. Of course you need to have a continually growing economy over time to make even it work, and that is a different sort of assumption.

Traders also get to choose their battles. Intelligence agencies don't. A trader can sit and wait for a market condition which fit his predictive trading models. He may only believe in his predictive powers 2% of the time, but as long as he stays out of the market when he doesn't have an edge he is likely to profit. If he misses a big move in a market, no big deal, things only matter when he decides to take a position. But a political analyst is forced to be "in the market" at all times. If he fails to foresee any major event, he has failed.

This does well in helping to explain why William White did far, far better than all of the leading academic macroeconomists in anticipating the current economic bust.

No TGS tag on this post?

For those interested in this topic, a good quick overview is this lecture by Tetlock: "Why Foxes Are Better Forecasters Than Hedgehogs" (you can find the CD on Amazon.com)

For more on consistently bad economist/financial forecasts (eg, analyst EPS estimates, economists predicting the course of the economy) see James Montier's "Behavioural Investing: A Practitioners Guide to Applying Behavioural Finance" (also on Amazon)

Forecasting as a separate topic of discussion exists for millions right now in Rotisserie baseball drafts all over the country. Each player shows up with a forecasting system's "cheat sheet" and battle it out, blog posts & chats debated the relevant systems, explain regression, standard deviations, etc. to people with no math background. Seminars are held in Arizona and at MIT.

The grandaddy of this group, the one most build derive from, Ron Shelton, has published 25 editions of Baseball Forecaster, filled with essays discussing forecasting methods and always includes one page of pithy cautions. (e.g., Christie Brinkley Law of Statistical Analysis: Never get married to the model", "Age is the only variable for which we can predict a rising trend with 100% accuracy"). This year his opening essay addresses "What have we learned over 25 years?". Answer: "No matter how much effort we put into playing this game, we are not nearly as smart as we think we are. We are all pretty much clueless."

Very good article, I like it, thank you

Dear Tyler,

Goodness: "modest but significant." The Supreme Court last week decided against a drug company hiding behind a "statistically insignificant" loss of smell from take a zinc compound to stop a cold. So it is now illegal to claim significance for merely statistical significance. Sorry, but the next time I catch you doing it, I'm calling a police officer!

Warm regards, Deirdre

Comments for this post are closed