Observers expect forecasters to show high confidence

Observers seem to focus on the target event and not its complement.  Bagchi and Ince have a new paper on this question:

Consumers routinely rely on forecasters to make predictions about uncertain events (e.g., sporting contests, stock fluctuations). The authors demonstrate that when forecasts are higher versus lower (e.g., a 70% vs. 30% chance of team A winning a game) consumers infer that the forecaster is more confident in her prediction, has conducted more in-depth analyses, and is more trustworthy. The prediction is also judged as more accurate. This occurs because forecasts are evaluated based on how well they predict the target event occurring (team A winning). Higher forecasts indicate greater likelihood of the target event occurring, and signal a confident analyst, while lower forecasts indicate lower likelihood and lower confidence in the target event occurring. But because, with lower forecasts, consumers still focus on the target event (and not its complement), lower confidence in the target event occurring is erroneously interpreted as the forecaster being less confident in her overall prediction (instead of more confident in the complementary event occurring—team A losing). The authors identify boundary conditions, generalize to other prediction formats, and demonstrate consequences.

Of course this also has relevance for the evolutionary processes governing pundits.

Here is a related press release (pdf).  For the pointer I thank Charles Klingman.


I predict that 90 percent of comments here will go off a tangent.

Omnipotence is underrated. Omniscience depends on it.

I support your prediction. Is Walker with his health care plan going to take the wind out of Trump?

Omnipotence means 100% chance of fathering a child, the opposite of impotence.

Well, this applies to GMOs pretty fine. Biology and plant scientists say GMOs are safe with a certain degree of confidence, Vandana Shiva says GMOs are absolutely dangerous (100% confidence)......Vandana Shiva wins :(

No, it doesn't. The paper is about events. The safety, or lack thereof, of a GMO is a fact, not an event.

Horizontal gene transfer from a GMOs to wild plants is an event. "Safety" is an umbrella for many events.

Horizontal gene transfer from conventionally bred plants to wild plants is an event too. One with equal probability.

Come on, I'm no GMO scaremonger. Horizontal gene transfer even if theoretically possible is extremely unlikely. Also, as HM said it is not restricted to GMOs, the conditions for it's occurrence exist since day zero of evolution.

The topic here is the "extremely unlikely" description of the HGT event made by scientists Vs the "it will absolutely happen" of activists.

What is the likelihood of Danny hitting the U.S. mainland? My observation about hurricanes is that they travel in much the same way as squirrels: not in a straight line. How does one predict the path of squirrels? If Danny's projected path is a straight line toward me, I can be confident that the forecaster has little knowledge of squirrels and that, fortunately, I will be safe. Of course, straight lines reflect our prejudice that future events will be the same as past events. Forecasters may not know squirrels but they do know past events.

Always give predictions based on a high probability framing. Got it.

And I heard it from the broadcasting master himself yesterday while flipping through the stations:

"I'm right 99% of the time" - Rush Limbaugh

until you predict their team will lose then a 30% prob is confident and good and 70% is bad.

Here's less academic version of this idea:

'Consumers routinely rely on forecasters to make predictions about uncertain events'

What a bizarre phrasing. For example, lots of people are interested in sporting event forecasts, but to call a sport team fan a consumer seems strange. And to call a gambler on sporting events a consumer of forecasts (in aggregate, as such forecasts will affect betting in many settings) seems equally strange.

Time to familiarize people with the concept of confidence bounds.

The consensus economic forecast is that the recovery will be weak. This is always the case at least in my experience going back to the 1960s. Of course, until now it has always been wrong because historically recoveries are always strong.

I've decided the reason econ0mist always forecast a weak recovery is that they quickly discover that no one believes forecast of strong recoveries.

Without commenting on past predictions, this time anyone who forecast a weak recovery was exactly right.

The asymmetry is obviously a flaw, but some of this may not be anomalous, depending on how the forecaster accounts for ambiguity and uncertainty and compounding and updating priors. A very diffuse prior might be a 50/50 chance, and increasing distance from equiprobability may signal the presence of additional information.

See also Philip Tetlock, foxes and hedghogs.

And dart-throwing chimps

Notice how Tyler tries to establish credibility by not titling his post, "Observers don't expect forecasters to show low confidence."

Comments for this post are closed