Philip Tetlock requests your help

He is one of the most important social scientists working today, and he requests that I post this appeal:

Prediction markets can harness the “wisdom of crowds” to solve problems, develop products, and make forecasts. These systems typically treat collective intelligence as a commodity to be mined, not a resource that can be grown and improved. That’s about to change.

Starting in mid-2011, five teams will compete in a U.S.-government-sponsored forecasting tournament. Each team will develop its own tools for harnessing and improving collective intelligence and will be judged on how well its forecasters predict major trends and events around the world over the next four years.

The Good Judgment Team, based in the University of Pennsylvania and the University of California Berkeley, will be one of the five teams competing – and we’d like you to consider joining our team as a forecaster. If you’re willing to experiment with ways to improve your forecasting ability and if being part of cutting-edge scientific research appeals to you, then we want your help.

We can promise you the chance to: (1) learn about yourself (your skill in predicting – and your skill in becoming more accurate over time as you learn from feedback and/or special training exercises); (2) contribute to cutting-edge scientific work on both individual-level factors that promote or inhibit accuracy and group- or team-level factors that contribute to accuracy; and (3) help us distinguish better from worse approaches to generating forecasts of importance to national security, global affairs, and economics.

There is more at the link and they even offer a small honorarium.


Best prediction: Everyone who enters will think they are above average in predicting.

And selection bias (among many other biases) would probably support that.

Who cares, as long as the study gets the best, the best, the best of you?

And that your models Learn to Fly.

I predict some people will enter just because they expect to do about average, just like everybody else.

But, that requires them to "know" what the true average of the group is, doesn't it? If they make an estimate of the true average and that estimate is lower than the true average, they enter.

The problem of overestimating your abilities is the flip side of underestimating your competitors.

I would like to participate

I predict you will be able to sign up to participate at the "this appeal" link at the top of this post.

It's very difficult to consistently make accurate predictions about fields for which exist fairly efficient markets and relatively little political correctness. Thus, beating the stock market is hard.

In contrast, it's easy to make more accurate than average predictions in fields in which political correctness distorts what is allowed to be said publicly. For example, I've been saying for years that the No Child Left Behind act would fail to make every public school student in America proficient in reading and math by 2014, and it would inspire large scale cheating on tests.

Also, predicting the stock market requires predicting the timing. When will No Child Left Behind collapse? Harder. Saying they won't reach the government goal by the stated government deadline doesn't count!

Hey, just because a prediction is a foregone conclusion doesn't mean it's not a good prediction!

Similarly, it's easy to predict accurately that mass immigration to the U.S. from poorer countries will continue to cause global carbon emissions to increase noticeably. The only plausible way this won't happen is if immigrants fail to assimilate economically so badly that most wind up using public transportation and living without air conditioning, or if mass immigration drives down the native birthrate very badly.

But, nobody wants to hear about predictions like this, despite their obvious importance.

I predict that mass immigration to the US from poorer countries will eventually improve the US soccer program.

The surprising thing is how it hasn't happened yet. The U.S. women's team in the recent World Cub was amazingly non-diverse for 2011: one Cameron Diaz-like half-Cuban blonde and one half-black woman raised (shades of Obama) by her white mother at the beach who had to learn about being black by majoring in black studies at Notre Dame.

The men's World Cup team has been about twice as black as Hispanic in 2010 and 2006.

And the quality of street food.

IIRC he was featured prominently in Gardner's Future Babble, which is something every politician should be tied down and forced to read.

I'd be honored to help without the honorarium, but I wish they'd be a little more explicit about the time commitment. I assume we will be part of a crowd.

How about playing Tetlock's game a different way:

We have many economists who profess to be able to "predict" the economy or the "effects" of certain public policy.

What if we pay, challenge or goad these prognosticators to participate in Tetlock's competition.

Do you think they would? How much would you have to pay them to put their reputation on the line? Would extreme predictions moderate if they were forced to participate in a competition and the results were posted.

Does this remind anyone of the Delphi system described by John Brunner in his novel Shockwave Rider. If I remember rightly, it was based on the premise that while nobody knows what is going on around here, everybody knows what's going on.

You mean ?

"will be judged on how well its forecasters predict major trends and events around the world over the next four years."

Wisdom of the crowds predicting trends? - most definitely, since crowds are what trends are all about.

Wisdom of the crowds predicting events? - who gets elected certainly, the stock market would be hard, maybe as a contrarian indicator but hardly specific predictions. Things like wether or not Iran is really developing the bomb, no chance at all since all anyone outside of a few spies and scientists really know is a regurgitation of the news. If the news is right they'll be right, if not they won't. Garbage in garbage out.

So in the end it all comes down to what they are trying to predict I think. Still, a worthwile project it seems to me.

The problem is that their solution to the problem is to essentially define the solution out of existence. A few selected individuals are not a "crowd". Improving the ability of individuals to predict the future will have very little effect at all on the ability of the group as a whole after the group grows past a certain size.

The goal is to identify ways to improve the accuracy of interpretation of emergent data from a crowd. That is essentially orthogonal to improving the ability of individuals to prognosticate.

Most of the events that we find interesting to predict are ones that are inherently hard to predict. Sometimes they are events that have been engineered to be hard to predict, like the winner of the Super Bowl.

In contrast, it's easy to make some predictions about some events. For example, I can predict to the second exactly when the sun will come up tomorrow. But that prediction is boring. I can also predict that Beverly Hills public schools will have higher average test scores than Compton public schools for each and everyone of the next four years. But, you find that prediction to depressing and annoying.

In summary, the "low hanging fruit" in predictions is to look for areas where systematic biases, such as political correctness, get in the way of making accurate predictions about important questions.

I think economists such as Paul Krugman, who trumpet their ability to predict accurately the future, should participate and make the results public.

I'll smoke all you poseurs.

Dear Professor Tetlock,
I believe your educational criterion is arbitrary. Speaking as a retired architect on behalf of many successful Americans who lack formal training past high school, I suggest that you have cut yourself off from what may be the most creative pool of talent in the country. This will certainly affect your results, just how cannot be predicted, except to say, that you no longer have a crowd.
William McCullam, American Institute of Architects, Emeritus
Newbury, Ohio

I actually encouraged a young friend of mine to apply (this seemed right up her ally, and she is a voracious browser) only to have her respond: "I finish my BA this Fall, two classes to go. Should I just lie and say I have it now? It's required qualification..."

My answer was, of course, no - don't want to corrupt the data - but it seemed a shame, hadn't noticed that part.

Her own comment on this

"nice comment, but really, it sounds like i'm a 22 year old kid. i'm a 30 year old woman with more schooling than many with graduate degrees, and it's just a sick joke of fate that i have no degree to show for it... :-/ "

I'm interested in participating and probably about as well-qualified as the average college graduate, but excluded by what seems like an arbitrary requirement. I see your point about getting a smaller range of opinions. It's possible that the degree requirement is a way of improving the odds of people staying with a four-year project.

Some of the other teams are open to those without college degrees, and the competition will be a good way to measure the differences. Watch for other announcements. (I represent one of the other teams. We expect to go public around August 20.)

Comments for this post are closed