AI dominates human peer reviews

We show that AI-generated article reviews are now better, by every measure we could construct, than reviews written by well-compensated human experts and judged to be of unusually high quality by journal editors and authors. We then discuss how this result may inform journal policy choices, with concerning downsides as well as potential paths to increase publication speed and quality; reduce adversarial interactions; improve the experience of editors, reviewers, authors, and readers; and advance science.

By Gary King and Steven Worthington, not even using what are now the very best models, rather based on Claude Opus 4.8.  The details of the dominance, as outlined in the paper, are striking.  It is not close.

On my Saez and Zucman post, I find it striking that some people are complaining all of a sudden.  Yet I have been offering, citing, and linking to commentary on published papers for twenty-four years now (and every single day).  Sometimes it might just be a casual offhand remark from me, whether positive or negative.  Typically this is simply taken in stride.  It is when you pull out the bazookas, and offer oodles of high-powered analytics, more astute than human referees, and combined with the ability to pursue follow-up inquiries, that people start getting worried.  Model that.

You can pick at individual criticisms from the AIs all you want, but the most damning observation is that the errors go in the ideological direction of the authors 97% of the time.  That is exactly what (the human) Phil Magness said years ago, based on an extensive canvassing of their work.  Basically we already knew that was true, and the AIs confirmed the human estimate.

On our side, we have taken steps, most of all the new MR archive, to make AI analysis of our work easier.  Soon you will be able to read on MR what the AIs think are our worst posts, respectively (Alex has prepared this).

In the meantime, I encourage the NSF, prize committees, universities, tenure committees, journals, and so on to do as many of their own AI-enabled audits as possible.  So far the world is not jumping at this opportunity.  You can model that too.