My Conversation with Philip Tetlock

Here is the audio and transcript, here is part of the summary:

He joined Tyler to discuss whether the world as a whole is becoming harder to predict, whether Goldman Sachs traders can beat forecasters, what inferences we can draw from analyzing the speech of politicians, the importance of interdisciplinary teams, the qualities he looks for in leaders, the reasons he’s skeptical machine learning will outcompete his research team, the year he thinks the ascent of the West became inevitable, how research on counterfactuals can be applied to modern debates, why people with second cultures tend to make better forecasters, how to become more fox-like, and more.

Here is one excerpt:

COWEN: If you could take just a bit of time away from your research and play in your own tournaments, are you as good as your own best superforecasters?

TETLOCK: I don’t think so. I don’t think I have the patience or the temperament for doing it. I did give it a try in the second year of the first set of forecasting tournaments back in 2012, and I monitored the aggregates. We had an aggregation algorithm that was performing very well at the time, and it was outperforming 99.8 percent of the forecasters from whom the composite was derived.

If I simply had predicted what the composite said at each point in time in that tournament, I would have been a super superforecaster. I would have been better than 99.8 percent of the superforecasters. So, even though I knew that it was unlikely that I could outperform the composite, I did research some questions where I thought the composite was excessively aggressive, and I tried to second guess it.

The net result of my efforts — instead of finishing in the top 0.02 percent or whatever, I think I finished in the middle of the superforecaster pack. That doesn’t mean I’m a superforecaster. It just means that when I tried to make a forecast better than the composite, I degraded the accuracy significantly.

COWEN: But what do you think is the kind of patience you’re lacking? Because if I look at your career, you’ve been working on these databases on this topic for what? Over 30 years. That’s incredible patience, right? More patience than most of your superforecasters have shown. Is there some dis-aggregated notion of patience where they have it and you don’t?

TETLOCK: [laughs] Yeah, they have a skill set. In the most recent tournaments, we’ve been working on with them, this becomes even more evident — their willingness to delve into the details of really pretty obscure problems for very minimal compensation is quite extraordinary. They are intrinsically cognitively motivated in a way that is quite remarkable. How am I different from that?

I guess I have a little bit of attention deficit disorder, and my attention tends to roam. I’ve not just worked on forecasting tournaments. I’ve been fairly persistent in pursuing this topic since the mid 1980s. Even before Gorbachev became general party secretary, I was doing a little bit of this. But I’ve been doing a lot of other things as well on the side. My attention tends to roam. I’m interested in taboo tradeoffs. I’m interested in accountability. There’re various things I’ve studied that don’t quite fall in this rubric.

COWEN: Doesn’t that make you more of a fox though? You know something about many different areas. I could ask you about antebellum American discourse before the Civil War, and you would know who had the smart arguments and who didn’t. Right?

And another:

TETLOCK:

…I had a very interesting correspondence with William Safire in the 1980s about forecasting tournaments. We could talk a little about it later. The upshot of this is that young people who are upwardly mobile see forecasting tournaments as an opportunity to rise. Old people like me and aging baby-boomer types who occupy relatively high status inside organizations see forecasting tournaments as a way to lose.

If I’m a senior analyst inside an intelligence agency, and say I’m on the National Intelligence Council, and I’m an expert on China and the go-to guy for the president on China, and some upstart R&D operation called IARPA says, “Hey, we’re going to run these forecasting tournaments in which we assess how well the analytic community can put probabilities on what Xi Jinping is going to do next.”

And I’ll be on a level playing field, competing against 25-year-olds, and I’m a 65-year-old, how am I likely to react to this proposal, to this new method of doing business? It doesn’t take a lot of empathy or bureaucratic imagination to suppose I’m going to try to nix this thing.

COWEN: Which nation’s government in the world do you think listens to you the most? You may not know, right?

Definitely recommended.