Gambling Can Save Science!
Some 25 years ago, our colleague Robin Hanson published Could Gambling Save Science?, one of the first papers explaining and advocating prediction markets. Since that time prediction markets have been used to better predict political events, Hollywood movie revenues, and corporate sales but perhaps oddly little has been done in the field that Robin initially proposed, science.
That has now changed. As part of the reproducibility project economists at the Stockholm School of Economics set up a market to predict which psychology papers would fail to replicate. From Ed Yong at The Atlantic:
Dreber’s experiment was born in a bar. Over drinks with her husband Johan Almenberg and roommate Thomas Pfeiffer, she was talking about an attention-grabbing psychological study that she thought was “cute, but unlikely to be true.” When she wondered how good her instincts were, Pfeiffer brought up another paper by economist Robin Hanson at George Mason University. Titled Could Gambling Save Science?, it suggested that researchers could get a more honest consensus on scientific controversies by betting on their outcomes, in the way that traders bet on the future prices of goods.
“It blew us all away,” says Dreber. In 2012, she and her colleagues contacted Nosek, who agreed to add prediction markets to his big Reproducibility Project.
The markets worked well and, verifying the wisdom of the crowds, they worked far better than any individual in the market. The authors of the study argue that prediction markets could be used more extensively in science:
[P]rediction markets are a promising tool for assessing the reproducibility of published scientific results. The prediction markets also allow us to estimate probabilities for the hypotheses being true at different testing stages, which provides valuable information regarding the temporal dynamics of scientific discovery. We find that the hypotheses being tested in psychology typically have low prior probabilities of being true (median, 9%) and that a “statistically significant” finding needs to be confirmed in a well-powered replication to have a high probability of being true. We argue that prediction markets could be used to obtain speedy information about reproducibility at low cost and could potentially even be used to determine which studies to replicate to optimally allocate limited resources into replications.