Testing peer review by running submissions through the process twice

In particular, about 57% of the papers accepted by the first committee were rejected by the second one and vice versa. In other words, most papers at NIPS would be rejected if one reran the conference review process (with a 95% confidence interval of 40-75%)

Here is another framing:

If the committees were purely random, at a 22.5% acceptance rate they would disagree on 77.5% of their acceptance lists on average.

That is from Eric Price on the NIPS experiment, there is more here.

For the pointer I thank a loyal MR reader.


Comments for this post are closed