A journal editor speaks about referees (and authors)

What are the most common errors that reviewers make when reviewing health papers for you?

There are three errors that reviewers make. First, many junior reviewers write really long reviews to show that they were thorough. This doesn’t help—if the paper has 8 problems then the editor is often most interested in the top two.

Second, some reviewers can also have really high standards in a way that creates lots of Type II errors—never accepting a paper. At the Review of Economics and Statistics, we were writing to accept more papers, but reviewers made this hard by using an impossible standard for identification.

Finally, and this is rare, but a by-product of the “triple-aim” (described above): some reviewers write reports with innuendo and meanness—I never went back to them and still think very poorly of these individuals. To be mean, when protected by the veil of an anonymous review process, is a deep pathology.

My advice is: write short reviews—don’t over referee or rewrite the paper—you are the reviewer, not the author. Be kind. Be kind. Be kind. Kindness is not the same as low standards, but posing questions and raising challenges with curiosity and humility. Always remember that an editor is reading the review, sharing it with other editors, and one’s nastiness is noted and remembered especially when directed towards a new member of the profession.

That is from an interview with Amitabh Chandra of ReStat.

Comments

Excellent advice.

Yes but I wish he talked about why academia is so nasty.

You have never heard of Sayre's Law? - "Academic politics is the most vicious and bitter form of politics, because the stakes are so low."

I am not real superb with English but I line up this very easygoing to interpret.

Respond

Add Comment

Respond

Add Comment

Respond

Add Comment

Respond

Add Comment

Re: the first point, what about thorough reviews intended as helpful advice to the writer, not as feedback to the editor?

Respond

Add Comment

It doesn't cost anything to be nice, Clockwork. You can catch more flies with honey than vinegar. Try it some time.

Who is Clockwork?

Respond

Add Comment

Let's examine your claim and see if it holds up to scrutiny. Let's first state an opposite proposition: It doesn't cost anything to be mean. And here's another contrary claim: It does cost something to be nice. I could argue for any of these 3 - how about you? Seems to me (never having reviewed a submitted paper) that the biggest "error" reviewers make is to review a paper which present conflicts of interest and/or bias issues for them. I don't know (and don't care enough to read/hear the original interview) what Chandra means by "we [editors?] were writing to accept more papers." I'd guess he means that the editors were requesting the author(s) make changes to those parts of their paper which reviewers found unacceptable in such a way as to make it acceptable. I don't have a clear idea of what an "impossible standard for identification" means - I suppose it means something about anonymization. I definitely have a nastiness streak. I try to muffle it but it escapes from time to time. Especially when writers make overly broad statements - for example implying they know or understand what the cost is for someone to reply to some risible claim that is seriously proffered by someone who should know better.

Excellent points. A man has to do what a man has to do. There is no reason to be polite or nice when someone makes an overly broad statement. No two-fisted American should have to put up what that. But would you dispute that flies are more attracted to honey than vinegar? That is what it's really all about. Cite your sources please.

Respond

Add Comment

Respond

Add Comment

Respond

Add Comment

We certainly can't do away with this "peer review" thing.

Some truth might get out, and the goyim might get alarmed.
Can't have that. Oy vey.

Respond

Add Comment

There are a lot of places in life where it's useful to play a role. The role of reviewer, the role of server, etc

In terms of people wigging out and being mean, that happens too. Mean reviewer, mean waitress.

I think we should cut them both some slack. It could be a hard week.

If you are having a hard week, hit the save button until you are in a better mood. One's hard week is not a license to ruin someone else's day.

Respond

Add Comment

Respond

Add Comment

This is similar to the advice of accomplished appellate lawyers. I clerked for an appellate judge before going to work at a law firm. Many of the briefs were atrocious, long and rambling, and raising ten or fifteen issues. Were those lawyers trying to persuade the appellate judges or impress their clients? The head of the appellate practice at the large firm I went to work for had three rules for briefs: raise no more than three issues, preferably only two, limit the length of the body of the brief to 12 pages, and don't criticize the judges below (to err is human).

Respond

Add Comment

I wonder whether there is some self-selection of reviewers towards reviewers that want to reject, be mean, and/or enjoy writing long reviews. Those that don't want to spend too much time writing long reviews or nitpick/criticize other people's papers might make more effort to decline being a reviewer.

Sometimes, one can even see the paper or abstract before deciding whether one wants to be a reviewer. I vaguely recall from my grad student days declining to review papers when I had a chance to preview them and they seemed really bad. Maybe, some people have the opposite tendency.

Respond

Add Comment

Respond

Add Comment

I see so many grad student papers these days, and papers by professors at non research institutions. The quality is truly abysmal, and editors seem to have refused to act as if they are still in the halcyon days of two decades ago, when submissions from foreign academics, grad students, and (effectively) non-research scholars were exceedingly rare. They no longer are, and good editors need to desk reject a lot more than they do.

When I do get a poor paper I know isn't going to get an r and r, but I think is grad student work, I often write these long reviews the editor dislikes. Not because the editor needs a long review (they already know it's a poor paper but don't like being "mean" and desk rejecting), but because clearly the student is not getting appropriate feedback from their advisors.

I think this is an important point: what are the goals of the review process and what are the responsibilities of the reviewer? Giving an evaluation to the editor is certainly one goal, probably the most important one.

But as you note, there are other important things that can be accomplished in these reviews. Such as giving somewhat massive feedback to an author who needs that feedback.

And as you state, there are alternatives: reject the paper without sending it out to a reviewer.

The article's advice seems to take too narrow a view, rather than accounting for these other goals and other ways that the publishing process could happen.

Respond

Add Comment

Respond

Add Comment

'some reviewers write reports with innuendo and meanness—I never went back to them and still think very poorly of these individuals. To be mean, when protected by the veil of an anonymous review process, is a deep pathology'

This process does not actually seem anonymous (by a normal use of the term, like anonymous voting) if the editor knows who the reviewer is.

The editor must know the reviewer because the editor chooses the reviewers and sends the paper to them.

Respond

Add Comment

The reviewer is anonymous to the author, not the editor.

Respond

Add Comment

Respond

Add Comment

The first is not an error but a feature: its normal even for good papers to have several minor issues, and being thourough is can be very helpful for authors.

Respond

Add Comment

" . . . some reviewers write reports with innuendo and meanness—I never went back to them and still think very poorly of these individuals. To be mean, when protected by the veil of an anonymous review process, is a deep pathology."

Surely the data can be assembled to show or at least suggest where such pathologically afflicted reviewers are perched: perhaps the most pathologically afflicted could even be identified properly.

We might then want to discover just what factors account for such pathology among academics and "professionals". (What contributions to pathology does tenure status confer? salary? academic lineage? e. g.)

Who begins tallying the data to show just how much "anonymity" in internet domains and venues contributes to widespread "pathological meanness"? Do social media platforms and internet venues breed "pathological meanness" intrinsically and inherently by providing for user anonymity?

What other factors might contribute to academic pathologies? Academic pathologies surely must include cognitive and epistemic pathologies as well as simple affective pathologies (once Holy Science equips us to attain "epistemic purity", all academics will presumably be required to submit to brain scans to reveal any evidence of brain lesions or neuropathies that could complicate their academic careers or those of their students.)

Respond

Add Comment

Having published scientific papers for 30+ years, I see a trend: reviewers are rejecting papers for what are increasingly minor issues that in the past would call for "minor revision". They also do not seem to know the boundaries of their own knowledge and make idiotic comments that show they did not understand simple English (maybe careless, maybe reading-challenged, maybe in a hurry). If they do not know what a Poisson distribution is or where Montana is (a European reviewer), they can go to Wikipedia instead of assuming that all readers share their ignorance and saying the paper is not clear.

With what I perceive to be ever greater internationalization of research and publishing, might it be a good idea to assume that some of your readers do not know where Montana is?

Even just making a comment on a blog, I'll often say "American football" instead of "football" because depending on who I think might be reading the blog and comments, a word such as "football" could have different meanings to different readers.

"Backpacking" is another such word. If it's mainly Americans reading the blog, we know what we mean when we use that word. To Brits and I think others in Europe, it means traveling via train, plane, bus, or hitchhiking with one's luggage all packed in a backpack in order to travel cheap and light. Nothing to do with hiking in the backcountry and camping.

Respond

Add Comment

Respond

Add Comment

My own experience, as recipient of referee reports on my work (but I’m far from alone in this observation), is that the error is more commonly in the other direction—reports are too short and often reveal that the referee didn’t take the time to read the paper carefully.

Who are these crazy referees who have the bizarre incentive to spend too *much* time on a report?

I can see why a journal editor, with an active research agenda of his or her own, might complain about reports not being even shorter than they already are and less of a burden to read. Suck it up. It’s the price you pay for having your name on a masthead.

And who are these snowflake authors who are hurt by “nastiness” in itself? Does anyone give a flying flamingo about tone? The real problem is that nastiness is employed by shirking referees as a low cost / low quality substitute for substance.

Relatedly, the double-blind system erects massive barriers to publication, especially to academics (or non-academics) from smaller institutions that have fewer resources to attend conferences. The “in crowd” from which most referees are drawn thinks to itself: “If this paper were any good, I would have seen it already.”

I’m not really complaining—this is how the profession works. But please, spare me the self-serving crocodile tears.

Respond

Add Comment

Respond

Add Comment