Is scholarly refereeing productive at the margin?

No, basically:

In economics many articles are subjected to multiple rounds of refereeing at the same journal, which generates time costs of referees alone of at least $50 million. This process leads to remarkably longer publication lags than in other social sciences. We examine whether repeated refereeing produces any benefits, using an experiment at one journal that allows authors to submit under an accept/reject (fast-track or not) or the usual regime. We evaluate the scholarly impacts of articles by their subsequent citation histories, holding constant their sub-fields, authors’ demographics and prior citations, and other characteristics. There is no payoff to refereeing beyond the first round and no difference between accept/reject articles and others. This result holds accounting for authors’ selectivity into the two regimes, which we model formally to generate an empirical selection equation. This latter is used to provide instrumental estimates of the effect of each regime on scholarly impact.

That is from a new NBER paper by Aboozar Hadavand, Daniel S. Hamermesh, and Wesley W. Wilson.  This is exactly the kind of work — critical, data-driven self-reflection about science — what Progress Studies wishes to see more of.

Comments

I could imagine an argument that the referee process is productive by making articles better in ways that don't generate citations yet still make the literature more transparent (and thus have spillover effects that generate citations for *other* articles).

At least some top mathematicians seem to believe that too many articles are published, too many resources are going toward writing bad or mediocre articles, and not enough resources are being devoted to making the top articles better. http://vlafforg.perso.math.cnrs.fr/files/prop-rapp-en-new.pdf

Respond

Add Comment

1) This result could be obtained by the longer review process allowing weaker articles to eventually be published (while they would be rejected by the single-stage review). Having weaker articles published rather than rejected is still a net benefit if they are good enough to have a positive impact in the field.

2) I'm skeptical of their ability to truly account for selection bias using the method mentioned, though I don't have access to the full article.

Another consideration is that scholars who have their articles rejected will often go on and submit to other journals until it is accepted.

This works against possible point 1) I made above, but it also undermines the claim about wasting review resources. Arguably, a similar amount of reviewers' time (or even more, because reviewers at different journals are starting over and making different points) is being spent on many articles in fields that have a single review standard.

Respond

Add Comment

Respond

Add Comment

"In economics many articles are subjected to multiple rounds of refereeing at the same journal, which generates time costs of referees alone of at least $50 million. "

There's no way they even create $50 in value. This is just cleverly disguised academic waste.

Respond

Add Comment

“Diversity means not bringing people with darker skin who use exactly the same models and ask exactly the same questions and reach the same conclusions,” said Cecilia Conrad, an economist who is now an executive at the MacArthur Foundation. “Embracing diversity means opening up to the kinds of new questions and new ways of seeing the world that will eventually improve economic science.”

I'd make the same comment about multiple rounds of refereeing: it promotes adherence to convention. With each round, the paper is likely to become increasingly conventional. It's like papers written by committee, reaching the lowest common denominator.

A problem in economics is the circular firing squad: in the context of research, it means finding some minor discrepancy or error in an effort to undermine the credibility of the research. And that's a function of the emphasis on empiricism, as if adding up the numbers somehow proves the reliability and value of the research.

That's not to say there aren't heretics in economics; indeed, some of Cowen's most favored economists are heretics (do I really need to name names). But heresy for heresy's sake proves what exactly?

Respond

Add Comment

I'm unable to access the article without paying for it. Can someone tell me how much longer the review and revision process is for econ papers than for psych, soc, and poli sci papers? Very curious. I've endured multiple revise and resubmits for the same paper at the same journal and found that the adjustments to the paper from rounds 2 onward were, at most, cosmetic. But they delayed the final decision on the paper by many months. I don't think editors should encourage two or three R&Rs; they are a huge waste of time, generally.

Respond

Add Comment

This is probably right, but shouldn’t be taken uncritically.

Citation can be an indicator of novelty of hypothesis instead of strength of results.

Pushback from past referees may have improved the authors ability to write and reason. So if every journal did this we would lose out.

Respond

Add Comment

Like dan1111, I am skeptical the authors have successfully found an instrument for selection bias regarding who choosed to go for the up-or-down systems. Did they use that old standby rainfall? :-)

As a longtime editor I agree with the general point and from Day One of my editing have striven to avoiid reviews beyond one rounds, with those that do go longer having it fairly clear that the paper is highly likely to be published if things can get fixed properly.

Going along with this effort has been a move over time to making lots of desk rejects to avoid wasting everybody's time. My observation is that the authors who complain about those are either very new and inexperienced or old egomaniacs who complain along lines of "How dare you desk reject ME???" Yeah, as any regular reader here should know, mangy old street fighter me dares to reject anybody and had rejected Nobel Prize winners (although I confess to never having desk rejected one of them, only after serious negative reports). So, I do have my limits on manginess and street fightingness.

But, again, I agree that this tendency in economics for many rounds of revisions at many journals is just a massive waste of time and effort, and frankly an embarrassment compared to how people in other disciplines view us.

Respond

Add Comment

I was unaware that Econ journals were so ... diseased. I guess it must reflect the Econ academic community. How many papers get more than 2 R&Rs? I can't think of a good reason not to simply reject any work if it's been rejected twice before and continues to be problematical. (And, yes, if there's still substantive (rather than cosmetic or style oriented) issues after the 1st review, the 2nd should either be pass or fail.) Otherwise, it just suggests the low quality of the research being produced...

Respond

Add Comment

Is there any real reason to referree stuff these days? And if a journal refuses to publish without it, who cares? I referreed lots of organic chemistry papers over the years, but the truth is that I could only be sure a result was valid by trying to replicate it.

Just publish everything, including attempts at replication being fully linked to the previous claims.

Respond

Add Comment

Here is a long to the article on one of the authors' website:
https://ahadavand.com/wp-content/uploads/2020/01/EISubmissionPaper1219.pdf

Respond

Add Comment

Respond

Add Comment