Peer review is younger than you think

by on September 17, 2017 at 12:03 am in Data Source, Education, History, Science, Uncategorized | Permalink

Via Ben Schmidt, the term becomes common only in the 1970s:

I’d like to see a detailed look at actual journal practices, but my personal sense is that editorial review was the norm until fairly recently, not review by a team of outside referees.  In 1956, for instance, the American Historical Review asked for only one submission copy, and it seems the same was true as late as 1970.  I doubt they made the photocopies themselves. Schmidt seems to suggest that the practices of government funders nudged the academic professions into more formal peer review with multiple referee reports.

Further research is needed (how about we ask some really old people?), at least if peer review decides it is worthy of publication.  Frankly I suspect such work would stand a better chance under editorial review.

In the meantime, here is a tweet from the I didn’t know she was on Twitter Judy Chevalier:

I have just produced a 28-page “responses to reviewer and editor questions” for a 39-page paper.

I’d rather have another paper from Judy.

By the way, scientific papers are getting less readable.

1 derek September 17, 2017 at 1:03 am

I’ll ask. I trudge through swamps with an older fellow looking for wildlife. He published some stuff years ago.

2 Demosthenes September 17, 2017 at 1:05 am

For “house” journals, the refereeing was often literally in-house.

E.g. Labor paper submitted to JPE? Gregg Lewis reviewed it.

3 Thanatos Savehn September 17, 2017 at 3:01 am

When the Left seized academia its first order of business was to build walls topped with ramparts; and peer review was among its first and longest-lasting. The unfolding reproducibility crisis in science is the realization that the Left was willing to sacrifice the lives of millions of cancer (and other chronic disease) patients over the last five decades in order to win a few political contests. They’re just the Borg though, and to the extent they know anything, they know only that they do. And the children of men never sleep too long.

4 ʕ•ᴥ•ʔ September 17, 2017 at 9:21 am

Meanwhile, the right simply went mad. As evidenced here.

5 Retinue September 18, 2017 at 12:30 am

The slightly-less-left has always been quite mad.

6 Is2la3 September 17, 2017 at 10:40 am

…the narrow discussion here is ‘Peer-Review versus Editorial Review’. What about other options?

Objective is to validate/verify research — what is best way to do that?

The flood of “research” since 1960 certainly has clogged the previous review system/norms… and the quality of current research is horrible — most of it loaded with methodological errors and is non-replicable.
(… and Thanatos has a valid point about academia political ideology influencing this situation)

Perhaps we need a basic group of Methodology-Checkers to initially screen new research papers … before papers are reviewed by the peer group of experts in that specific field of research. This would reduce the specialist peer-review workload by 80% … filtering out the cr@p. These ‘Methodology Checkers’ would need expertise only in statistical methods and basic research procedures… and thus could screen papers in any area. This would be akin to running a Spell-Checker on a submitted paper, before deeply analyzing content.

The current “review system” obviously is not working.

7 Retinue September 18, 2017 at 12:28 am

The only system capable of handling so much crap is a sewer system.

It’s time to “dump” “publish or perish” in the dustbin of history where it belongs, along with the bureaucrats, guardians of the realm, and their fake science.

8 Hua Wei September 17, 2017 at 11:09 am

Yeah, the rampart-induced cancer crisis. File unser We demand more asbestos! school of rhetoric. Seriously, for all the whinning about gow Al Gore or gays or faminists are to blame for the Right nominating Trump and voting for him , we have every evidence Trump is exactly what they are, just a little bit more successful at scamming people.

9 Retinue September 18, 2017 at 12:23 am

Science is apolitical; Al Gore is so last decade, and gays and feminists are out — and trannies are in. Get with the times, man.

10 ttt September 18, 2017 at 5:37 pm

“sacrifice the lives of millions of cancer”

wont somebody think of the cancer!!!

11 Boris_Badenoff September 17, 2017 at 4:19 am

It is worth noting that every single major case of scientific fraud in the last 40 years has been peer-reviewed & published in a respected journal. This includes the fellow who published an average of one paper ever six weeks or so, reflecting work that typically takes about 18 months to complete. How could no one have caught on, just by that super-human output, if not by serious reviews?

The tendency of reviewers, especially of works by well- known and/or senior authors, is to confirm. Going along entails little risk while objecting rocks the boat. Basic CYA for career insurance.

12 CD September 17, 2017 at 12:28 pm

1. As Loki says reviewers don’t repeat experiments or re-run regressions. Peer review addresses an important but narrow band of concerns.
2. A lot of top journals have rejection rates around 90%, though some of that happens before papers get sent out for review.
3. FWIW my own reviewing seems roughly evenly divided betwen reject and revise-and-resubmit recommendations. Earlier this year I made my first first-round accept recommendation ever.
4. If CYA is a goal you are more likely to reject. When there is mischief in the ed process, positive or negative, it is usually in the editor’s choice of reviewers.

13 Loki September 17, 2017 at 4:49 am

Boris_Badenoff – its very likely that major cases of scientific fraud involved peer reviewed publications. It would be difficult for a case to be major if it hadn’t.

However, discovering fraud really isn’t what peer review is meant to be doing. To uncover fraud someone would usually need to check to see whether the methods described in the paper were actually carried out by the authors. A usually anonymous reviewer in another institution can’t do that. The people best placed to uncover fraud are the fraudster’s colleagues (as was the case with, for example, Diederik Stapel

As for the OP, yes, my impression is that in the old days editorial review was the norm, and sending articles out for review by peers happened sometimes but inconsistently.

The overall context is that in the social sciences in North America and Western Europe at least there was a huge expansion in the number of academics in the 60s and 70s, and that coincided with a huge expansion in the number of journals. So the way that research was conducted was very different. Today’s double blind peer review practices are very difficult now in niche areas where there are a handful of people with expertise and everyone knows each other. Back in the old days that would have applied to much wider areas. Easier back then to just admit that the journal editor is best placed to make an assessment of what peer review is meant to focus upon: are the methods a sensible way of addressing the hypothesis or research question; does the article make a significant contribution to the field; are the methods and findings written up properly; has the author demonstrated knowledge of other existing work etc.

Though of course there is the criticism that in the old days getting an article accepted was as much about being friends with the right people as the quality of the research.

14 Retinue September 18, 2017 at 12:20 am

The quality of the research, or the quality of the science?

What a coincident ascendance of “research”.

“The overall context is that in the social sciences in North America and Western Europe at least there was a huge expansion in the number of academics in the 60s and 70s…”

15 Philip George September 17, 2017 at 6:12 am

At least in physics the practice began in Einstein’s lifetime.

I remember reading that he was extremely upset that his paper had been sent outside for review. Before that the editor used to do the reviewing alone.

I did a search and found a result:
It is entirely on peer review.

16 Bill Benzon September 17, 2017 at 7:02 am

Here’s another piece on the same topic:

Indeed, for most of the history of scientific journals, it has been editors – not referees – who have been the key decision-makers and gatekeepers, to the extent that journals were often known colloquially as “Editor X’s journal”. At Nature, for example, editors were firmly in charge until the mid 20th century. Whether this practice could be described as peer review depends on whether we adjudge the editor to be a “peer”. For many scholarly editors, this has, of course, been true. But it’s not really what we mean by peer review now.

At early Royal Society meetings, research findings were presented, often demonstrated and frequently discussed. But while it is possible to say that this means they had undergone scrutiny by well-informed scholars, that could be deemed to be peer review only to the extent that material presented nowadays at workshops and conferences (or on preprint servers) can be said to have been peer-reviewed.

17 dearieme September 17, 2017 at 7:46 am

A relevant question is: what is the purpose of a peer review? People seem to assume it means that the reviewer is giving a thumbs up to the content of the paper. WhenIwasbutalad, however, I was instructed that my purpose was to assess the degree to which the paper was complete enough, and clear enough, that the reader could form his own judgement on its content. In other words my principal enemies were not error and mendacity but muddle, ambiguity, verbosity, missing explanations and absent facts. Naturally if I noticed an error I’d point it out, but that was just a bonus.

18 rayward September 17, 2017 at 7:48 am

“Peer review” of papers submitted to a law review consists mainly of reading the cited authorities to confirm the accuracy of the proposition for which they are cited and of updating (what was called “shepherdizing”) the authorities to confirm that they haven’t been overruled or otherwise modified. This is typically done by top law students who are “on law review”. As for concepts, legal reasoning, etc., that is typically reviewed by the faculty advisers to the law review. This process of “peer review” has existed as long as their have been law reviews.

19 Retinue September 18, 2017 at 12:15 am

Which is not all that much longer than peer review, apparently.

20 AnthonyB September 17, 2017 at 8:52 am

Here is a detailed discussion of Einstein’s reaction to being peer-reviewed at Physical Review:

21 Bill September 17, 2017 at 9:06 am

Sounds to me that a person who criticizes peer review just had a paper returned to him with comments.

As for peer v. editorial review (i.e., a single editor) I am in favor of peer review because part of the process is obtaining the suggestions and comments of many others, whether you like it or not. An editor is too busy.

22 Retinue September 18, 2017 at 12:12 am

“Sounds to me that a person who criticizes peer review just had a paper returned to him with comments.”

Work on your reading comprehension.

Or, is the recency of peer review not, in your opinion, sufficient evidence that His truth is marching on? Personally, I believe in Progress; it’s the hip new thing.

Too bad science has stalled, its lifeless corpse reanimated as Science™, like the Bug from Men In Black. Dude, where’s my flying car?

23 Evan Osborne September 17, 2017 at 9:10 am

I have a book coming out early next year from Stanford University Press (n self-regulating social systems) where I devote a chapter to the history of the scientific method. Peer review as we now know it got off the ground in the 1950s. Carbon paper was part of the story.

24 DanC September 17, 2017 at 9:26 am

I would like to see Sam Peltzman comment on this issue

25 DanC September 17, 2017 at 9:49 am

Political Influence Behind the Veil of Peer Review: An Analysis of Public Biomedical Research Funding in the U.S.

Theory, Evidence and Examples of FDA Harm

Theory, Evidence and Examples of FDA Harm

26 DanC September 17, 2017 at 9:51 am

sorry link to second article

27 Ray Yang September 17, 2017 at 9:53 am

We should probably look for natural experiments — fields where prominent journals stayed with editorial review for a good long while, or maybe are still doing it, to see the difference. I suspect peer review has something to do with volume of submissions. A lot of profs I know simply could not handle the current volume of submissions.

Ironically enough, I’ve heard stories where the current volume of submission is so high most papers are given rejections before being sent out by review, and you need to have an in with the editors just to get your paper refereed…

28 Caipirinha September 17, 2017 at 9:59 am

This is true. I think usually the editor reads the abstract before sending it to review. Its a very SUBJECTIVE process I would like to note.

29 Jonathan Dingel September 17, 2017 at 11:25 am

I believe that review by a team of outside referees has a long history in economics. Take the famous story in which Debreu was one of the Econometrica referees for McKenzie’s submission of his contemporaneous paper in 1953. There are lots of (famous) people involved in the process. Solow asks Hurcwiz and Nash for reports; when Nash doesn’t referee, Solow gets a report from Debreu. This is way before 1970.

30 jseliger September 17, 2017 at 11:56 am

My impression too (perhaps wrong) is that if you ask someone to review a paper, they perceive their job as finding something wrong with it and sending it back for R&R. Which leads to “too many cooks spoil the paper” and other problems.

31 CD September 17, 2017 at 3:01 pm

MMV, but your job is to help the journal in its project, which is building the lit in some area. Breaks down into (a) gatekeeping (b) helping along work that’s rough but potentially important (c) occasionally, helping novice scholars learn mechanics. There’s actually a slight disincentive to R&R, because you commit to dealing with revisions.

The surplus cooks problem happens, but is usually the fault of the editor. IOW when you get a dog’s breakfast of conflicting referee reports on your paper, the editor should provide guidance on what you need to do.

But normally, an R&R on your paper is fabulous news! It means a high chance of getting published plus professional guidance on making the paper better.

32 Li Zhi September 17, 2017 at 7:53 pm

Anyone know whether the graph is appropriate? Word distributions (English) typically follow Zipf’s Law which is a power law distribution. I’m just wondering if the ordinate shouldn’t be on a log scale…

33 TGGP September 18, 2017 at 12:45 am

I remember The Journal of Medical Hypotheses was editor-reviewed, but Elsevier shut it down in reaction to backlash from them publishing a Duesberg paper.

34 Loki September 18, 2017 at 4:33 pm

Medical Hypotheses is still up and running: But after the Duesberg paper they Elsevier changed the editor an made the journal send articles out to peer review.

Comments on this entry are closed.

Previous post:

Next post: