The robustness reproducibility of the American Economic Review

We estimate the robustness reproducibility of key results from 17 non-experimental AER papers published in 2013 (8 papers) and 2022/23 (9 papers). We then subject each robustness report to two independent, expert reviews. Including robustness tests rated as equally or more valid than the original analyses by expert reviewers, the fraction of significant robustness tests (p<0.05) varies between 0% and 93% across papers with a mean of 51%. The mean relative t/z-value of our robustness tests varies between 11% and 152% with a mean of 70%. Surveyed economists overestimate robustness but are able to predict which papers are most robust.

That is from a new paper by Douglas Campbell, Abel Brodeur, Anna Dreber, Magnus Johannesson, Joseph Kopecky, Lester Lusher, and Nikita Tsoy.  Here is very useful Twitter coverage from Douglas Campbell.