How replicable are economic experiments?

In a paper just published in Science, Colin Camerer of the California Institute of Technology and a group of colleagues from universities around the world decided to check. They repeated 18 laboratory experiments in economics whose results had been published in the American Economic Review and the Quarterly Journal of Economics between 2011 and 2014.

For 11 of the 18 papers (ie, 61% of them) Dr Camerer and his colleagues found a broadly similar effect to whatever the original authors had reported. That is below the 92% replication rate they would have expected had all the original studies been as statistically robust as the authors claimed—but by the standards of medicine, psychology and genetics it is still impressive.

One theory put forward by Dr Camerer and his colleagues to explain this superior hit rate is that economics may still benefit from the zeal of the newly converted. They point out that, when the field was in its infancy, experimental economists were keen that others should adopt their methods. To that end, they persuaded economics journals to devote far more space to printing information about methods, including explicit instructions and raw data sets, than sciences journals normally would.

For the pointer I thank John B. Chilton.