Rejection is Good for Citation (maybe)

A paper in Science covering over 80 thousand articles in 923 scientific journals finds that rejected papers are ultimately cited more than first acceptances.

We compared the number of times articles were cited (as of July 2011, i.e., 3 to 6 years after publication, from ISI Web of Science) depending on their being first-intents or resubmissions. We used methods robust to the skewed distribution of citation counts to ensure that a few highly cited articles were not driving the results. We controlled for year of publication, publishing journal (and thus impact factor), and their interaction. Resubmissions were significantly more cited than first-intents published the same year in the same journal.

The author’s argue that the most likely explanation is that peer review increases the quality of manuscripts. Rejection makes you stronger. That’s possible although the data are also consistent with peers being more likely to reject better papers!

Papers in economics are often too long but papers in Science are often too short. Consider the paragraph I quoted above. What is the next piece of information that you are expecting to learn? How many more citations does a resubmission receive! It’s bizarre that the paper never gives this number (as far as I can tell). Moreover, take a look at the figure (at right) that accompanies the discussion. The authors say the difference in citations is highly significant but on this figure (which is on log scale) the difference looks tiny! This figure is taking up a lot of space. What is it saying?

So what’s the number? Well if you go to the online materials section the authors still don’t state the number but from a table one can deduce that resubmissions receive approximately 7.5% more citations. That’s not bad but we never learn how many citations first acceptances receive so it could be less than 1 extra citation.

There’s something else which is odd. The authors say that about 75% of
published articles are first-submissions. But top journals like Science and Nature reject 93% or more of submissions. Those two numbers don’t necessarily contradict. If everyone submits to Science first or if everyone never resubmits to Science then 100% of papers published in Science will be a first submission. Nevertheless, the 93% of papers that are rejected at top journals (and lower ranked journals also have high rejectance rates) are going somewhere so for the system as whole 75% seems implausibly high.

Econ papers sometimes exhaust me with robustness tests long after I have been convinced of the basic result but Science papers often leave me puzzled about basic issues of context and interpretation. This is also puzzling. Shouldn’t more important papers be longer? Or is the value of time of scientists higher than economists so it’s optimal for scientists to both write and read shorter papers? The length of law review articles would suggest that lawyers have the lowest value of time except that doesn’t seem to be reflected in their consulting fees or wages. There is a dissertation to be written on the optimal length of scientific publications.

Comments

I am reminded of a colleague once describing the two principal journals in our field, fluid dynamics. Of the one, he said, "It's just the voice of the establishment printing what everyone already believes." The other journal was a better place to find something you hadn't already thought of.

Perhaps once-rejected, more-highly-cited papers are the ones that have more novel things to say and a shakier basis for saying them.

The Quillette website is currently running an article by a professor who had a paper (some sort of conjecture about the mathematics underlying the Greater Male Variance hypothesis in intelligence research) accepted and then simply made to disappear, without being either published or rejected, after a political outcry. This may be the model for the future of "challenging" scholarship.

They're not just rejected but they are rejected and later accepted. Maybe rejected because they upset the status quo and later accepted articles because they have new findings. New findings will make them more likely to be cited.

The students who lack verbal facility, don't like writing papers, become scientists. The students who have verbal facility, and find pleasure in endless talking or writing, become lawyers (or at least law professors), and write articles of endless length. (The ones at the very top become Supreme Court clerks or judges, and write interminable opinions.)

"The length of law review articles would suggest that lawyers have the lowest value of time except that doesn’t seem to be reflected in their consulting fees or wages." It's always safe to insult lawyers. The lawyers who write most law review articles are not practicing lawyers but students or academics - practicing lawyers don't have the time to write serious law review articles. Here is an article about a law review article on antitrust law written by a law student and published in the Yale Law Journal in 2017 that has become something of a sensation, as has the author, Lina Khan, because the article proposes an entirely new way to consider antitrust law in the era of Amazon, Facebook, Google, etc.: https://www.nytimes.com/2018/09/07/technology/monopoly-antitrust-lina-khan-amazon.html. The link to an online copy of the law review article is in the NYT article. As for the length of law review articles, one has to remember that lawyers are trained as advocates, and many believe that advocacy requires length (what I refer to as the shotgun approach to advocacy). I'm a firm believer in brevity, something I came to appreciate while reading book length legal briefs while clerking for a judge, later confirmed by the head of appellate practice at my first law firm who had a rule for appellate briefs of no more than three issues and no more than 12 pages in length. Judges are real busy and won't read the tomes lawyers often submit. So why do most lawyers submit lengthy briefs? For the reason Tabarrok states in his criticism of lawyers: fees. Clients expect their lawyers to write the tomes, and lawyers, mindful of their fees, satisfy their client's expectations by charging by the pound.

If you submit a paper multiple times, then more people know about your paper because they reviewed it at some stage, making it more likely that these extra reviewers will later cite your paper.

Yes I was thinking of this, plus being still in the review stage it might be more likely to be presented at few more times as well.

As far as optimal length goes, Science/Nature/Cell papers often have long supplemental materials. Often you want the short version to reach many people, and the long version to be critical for a small number of people. The supplement let’s you do both at once.

In medicine, the journals limit the length (words/figures/tables/pages) of the article, which often results in 10 ish page articles sometimes with 100+ page of supplementary appendices. My specialty's journal (Neurology) recently switched to publishing new research in one-page short forms with full length papers and appendices online.

"The length of law review articles would suggest that lawyers have the lowest value of time except that doesn’t seem to be reflected in their consulting fees or wages"

It is reflected in both of them.

>That’s possible although the data are also consistent with peers being more likely to reject better papers!

Or more controversial.

'but Science papers often leave me puzzled about basic issues of context and interpretation'

We knew that.

'Shouldn’t more important papers be longer?'

Maybe you are unfamiliar with this paper, which is actually quite short, at least when available in HTML on the Internet - 'On the Electrodynamics of Moving Bodies.' Admittedly, Einstein is no Knausgaard, so you cannot judge the importance of the scientific work using a word count. https://www.fourmilab.ch/etexts/einstein/specrel/www/

'There is a dissertation to be written on the optimal length of scientific publications.'

Not in physics - there, the equations tend to be the important thing, not the words. Though it did take Einstein four papers to handle general relativity - using the original edition page counts found in wikipedia, a total of 20 printed pages, which is even less than his decade earlier special relativity paper, which was 30 pages in print.

The graph and its discussion are awful in a way that is sadly commonplace. As you correctly note, the interesting question is "how big is the effect?," not "what's the p-value?" Unfortunately, the latter is what is routinely stressed, especially in social science and (distressingly) much of biology. Whether this is because the authors
(i) don't understand that a vanishingly small and totally unimportant effect will have as small a p-value as one wants for a large enough dataset,
(ii) want to get a splashy but meaningless paper in Science by simply stating "one group is different than another group", without realizing that every set group is different from every other group, or
(iii) have run out of meaningful effects to explore but have to put out *something*,
I don't know.

I was just corresponding with a friend and colleague about this toic yesterday, not related to this paper. It's amazing how many people realize this "p-value approach" is a problem, yet we can't figure out what to do about it.

"(i) don't understand that a vanishingly small and totally unimportant effect will have as small a p-value as one wants for a large enough dataset"

You are confusing about "effect size" (beta) and "statistical significant" (p-value). The two are not the same, and both are important. The abundance of non-replicable results are the consequence of ignoring or abusing p-value. Effect size in one experiment does not reflects replicability whereas p-value is. On the other hand statistical significant result with minute effect size might not worth to be border with.

"without realizing that every set group is different from every other group"

You do not seem to understand the function of statistical analysis.

"but have to put out *something*"

Aren't those non-replicable "large" effect papers?

https://faculty.washington.edu/agg/pdf/Gwald_Gonz_Har_Guth_Psychophys_1996.OCR.pdf

"Effect sizes and p values: What should be reported and what should be replicated?"

"Editor's note. This paper was invited by the Board of Editors of Psychophysiology to address standards for reporting data and replication in psychophysiological research. The perspectives that were developed are not specific to psychophysiological data but rather apply to the analysis and interpretation of various forms of empirical data. Although the authors' recommendations regarding statistical methodology might be regarded as controversial, it is our hope that the examples and recommendations offered in this article will focus attention on important issues concerning replicability in the field."

I have not looked at the article, but the results do not seem surprising to me. Authors tend to submit articles to journals in order of impact. Their initial submission tends to depend on the author's own estimate of the likelihood of acceptance which is typically their own estimate of the quality of their paper. Consider the following (1) An author submits a paper to higher impact journal X, the paper is rejected and then is submitted and accepted at lower impact journal Y . (2) The same author submits another paper initially to journal Y because that is the best the author expects to do. It should not be surprising that the first paper will have more citations (a measure of quality) on average.

To clarify point 2 it should be read as follows: (2) The same author submits another paper initially to journal Y because that is the best the author expects to do AND JOURNAL Y ACCEPTS THE ARTICLE.

Yup, I think this is the best explanation for the phenomenon.

Rejection builds character.

Since there are almost nine rejected papers for each accepted paper, shouldn't there be about nine times citation to the rejected papers? Those who cite papers don't know if the papers were accepted or rejected. Also, I would guess that papers which comport with conventional wisdom are more likely to be accepted: reviewers are more likely to ask for more support for papers that don't. If that's true, then it's understandable that papers which don't comport with conventional wisdom would be cited more often. Why cite a paper that concludes that Trump supporters are exceptionally tolerant of people unlike themselves.

I suppose if papers and submisdions are rejected because of political correctness, the science behind them will still hold up. The history of science is a battle between the orthodox and dissenters. Accidental discoveries have resulted in major breakthroughs. Yet, in the long run, we still have much to learn. Political correctness and orthodoxy often delays our understanding. Thankfully the internet breaks down many inhibiting barriers to discovery

1) I know little about scientific publishing, but how do can you count the number of times a paper is first submitted (to different journals)? I know that some (if not most) journals require some sort of exclusivity agreement, but I don't know the nuts and bolts of that. Clearly, having a a report on X rejected doesn't preclude (infinitesimal) alteration to the paper and "new" submission elsewhere. Different journals have different "styles" so alterations have to be the norm. I don't see how such agreements are enforceable. 2) The plot indicates the median at about 8 citations for the "1st`s" and by simple deduction that means the increase in count is ~0.6 citations/paper (7.5%, i.e. laughable). (As an aside, it might be informative to determine whether there's any difference in the citations from the REVIEWERS lab for papers which have been judged 'redeemable' and if this effect is significant.) 3) Science mag. has been overly brief for decades, it would make sense IF they enforced their own policy of requiring sufficient material be available to allow the replication of the work (found either as part of the submission, or on-line/3rd party, or from the authors), but they don't.

> 2) The plot indicates the median at about 8 citations for the "1st`s" and by simple deduction that means the increase in count is ~0.6 citations/paper (7.5%, i.e. laughable).

You have to look at the context. It is generally assumed that re-submitted papers are of lower quality. Instead they are of comparable quality and even statistically have more citation than the '1st intent' papers. From digitising the chart,

log(M1st) = 0.916, M1st = 8.23

log(Mresub) = 0.964, Mresub = 9.20

Mresub/M1st = 111.74%, about 12% better.

"Econ papers sometimes exhaust me with robustness tests long after I have been convinced of the basic result but Science papers often leave me puzzled about basic issues of context and interpretation. " - this could be due to economic researcher's need to develop their own tests to draw conclusions; in order to prove their effectiveness, they need to be rigorous about their accuracy. In basic science and medical research, it may be that results don't have to be run through as thorough a process of analysis in order to draw conclusions. When the research is targeted at a highly specific subset of the research community, even the explanation of why results are valid can be left out.

Robustness tests are a perfect example of material that should be primarily in an online archive only.

The readership is likely divided between people who really care about this research and want to see all the gory detail of robustness, and people who would be satisfied with some summary statements about robustness checking.

A general publishing strategy is to submit to the highest impact journal you think has a chance, then go to a lower tier journal if rejected. So the fact that a paper was rejected is generally evidence that its authors thought there was a chance of it being accepted at a higher tier journal, and you would expect it to be better than average for the lower tier journal.

Econ papers have indeed become way too long, with the circus of endless robustness tests having become just that, a circus.

As it is, the AER, which has become a major offender in this, is now ranked only 12th on the recursive discount impact factor list of RePec that many now consider to be the most serious journal ranking measure.

In my field (Plasma Physics) most papers are short because:

- a paper is written to speak to a single, narrow fact. We work in the gaps, and those are many but tiny.
- we expect to learn by working the math ourselves whilst reading. Math is denser than prose.
- if the paper says something useful, we invite the author to give a talk. Or we send a grad student to them to learn more, etc.

Papers serve a different purpose in physics.

Economics papers have plenty of math that takes plenty of work. Still econ papers are often huge, most recent papers are between 50 and 100 pages long. I think that the huge size of econ papers is due to the fact often econ papers tends to be whole models of an economy with very high level of detail and the latest research often consists of more general and complex versions of older models with even greater levels of detail.

Economics is converging back to books, where academic economics began and apparently it will be where it will end.

Isn't it obvious? Research represents an aggression against the existing edifice of knowledge. The best papers are the ones that hurt the status quo the most, the papers most likely to be rejected are not only the "bad" papers but the papers who are the most innovative and original. The stuff that gets accepted easily is the stuff that is just regurgitating the existing edifice of knowledge and so is not contributing anything that is really original.

Comments for this post are closed