What happens when we correct for publication bias?

There is a recent paper by Leif D. Nelson, Uri Simonsohn, and Joseph P. Simmons on this topic, the abstract is this:

Journals tend to publish only statistically significant evidence, creating a scientific record that markedly overstates the size of effects. We provide a new tool that arrives at unbiased effect size estimates while fully ignoring the unpublished record. It capitalizes on the fact that the distribution of significant p-values, p-curve, is a function of the true underlying effect. Researchers armed with only the sample sizes and p-values of the published findings can fully correct for publication bias. We demonstrate the use of p-curve by reassessing the evidence for the impact of “choice overload” from the Psychology literature, and the impact of minimum wage on unemployment from the Economics literature.

When it comes to both the choice overload effect and the minimum wage, correcting for publication bias implies a lack of significance in the overall tenor of the results.  In passing I am not sure the minimum wage is the best example here, since a “no result” paper on that question seems to me entirely publishable these days and indeed for some while.

For the pointer I thank Kevin Lewis.  And Kevin Drum adds comment.


Comments for this post are closed