Since the internet, do fewer researchers read research papers?

by on February 20, 2017 at 12:42 am in Economics, Education, Science, Uncategorized | Permalink

The obvious equilibrium is that more researchers can download papers from the internet, and thus we expect more papers to be read by a greater number of people.  If lay people enter the calculus, this is almost certainly true.  But what about researchers?  I am not convinced that more reading (of each paper) goes on, or that it should go on.

Most people, including researchers, cannot easily figure out if the main result of a research paper is correct.  That is true all the more as time passes, because the mistakes become less and less transparent.  But they can figure out who can figure out if the paper is right, and sample that opinion.  The internet aids this process greatly.  For instance, it is easier for me to find out what Bob Hall (one of the great paper analysts/commentators of all time) thought of a macro paper, if only by using email.  If I can find out whether or not the paper is true, often I don’t have to read that paper, though I may go through some parts of it.  The internet also gives me access to better summaries of the paper, if only in parts of other papers.

In this sense, researchers may rely on a fairly thin substructure of evaluation, though one of increasing accuracy.  As science progresses, perhaps scientists do/should spend more time honing their research specializations, and less time reading papers they are not expert evaluators for.  They do/should spend more time reading the papers where they are the expert evaluators, but that may mean reading fewer papers overall.

Viewed as a productivity problem, perhaps your read is competing against “further spread of the read and evaluation from the best expert” and is losing.  Efficient criticism is also sometimes winner take all.

I am indebted to Patrick Collison for a conversation on this topic, though of course he is not liable for any of this.  Neither he nor I have read a paper on such matters, however.  Thank goodness.

1 William February 20, 2017 at 1:41 am

I emailed my friend to verify if this blog post is true. He replied that truth is rarely pure and seldom simple.

2 Bill-me February 20, 2017 at 2:25 am

I like that. In my view the only route to truth is your own research. One drawback … no time to write papers.

3 prior_test2 February 20, 2017 at 2:07 am

“Mr. Watson — Come here — I want to see you.” – IBM

4 dan1111 February 20, 2017 at 2:37 am

How do you figure out that you can trust Bob Hall? Or find the Bob Hall for your discipline?

A big problem with science is that a lot of it boils down to appeals to authority. It’s no wonder that people distrust science–they are expected to simply take it on faith that designated experts are correct about what the evidence says. And then the lay people get blamed for being “anti-science”, “science deniers”, etc. I think science needs to come up with better ways of presenting and evaluating evidence.

5 Veobaum February 20, 2017 at 11:19 am

Yes and no. I imagined Bob Hall giving the Tyler the 3 to 5 bullets of a papers premise and problems and his subjective conclusion rather than just a “Thumbs up from on High”.

The point is about finding the right couple of people who have read the paper critical and can provide clear arguments that you can then consider.

6 anon February 20, 2017 at 12:23 pm

Why would you pick science, which has the most transparent claims, and for that reason top reproducibility?

http://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970

7 JWatts February 20, 2017 at 2:33 pm

” And then the lay people get blamed for being “anti-science”, “science deniers”, etc.”

That only becomes significant when the science becomes highly politicized.

8 TMC February 20, 2017 at 3:47 pm

And in true Orwellian fashion, those who question the results are anti-science.

9 Hazel Meade February 20, 2017 at 11:03 pm

Oh, it can be true even when it’s not politicized at all.

10 Axa February 20, 2017 at 3:17 am

As PhD student must analyze articles, become a new expert on the topic and find out if it is true or not. Blindly following the advice of evaluation structures is not an option.

11 Yoav February 20, 2017 at 3:19 am

As someone who had read many papers, I can say that most of them are:
1) very small in scope. Meaning that they propose very small increments in knowledge, even if the work they did is high.
2) Lots of garbage. Meaning something is not relevant / not feasible/ not interesting/ wrong etc.
3) InAccesibility : There are lots of differnt journals, and the university I am in don’t have access to all.
4) Searching – it can be hard to find relevant things.

This makes it that when you look, you look at subset, and a lot of time it is garbage/ not relevant to you.
In the end you go by word of someone you know and trust on which papers to read.

12 Axa February 20, 2017 at 4:11 am

Short Vs long term issue. In the short term is may seem more efficient to get advice. It frees time to do other things. However, university time is good to waste time reading lots of garbage, small increments or developing searching skills. It may feel like a waste of time (not publishing) but you’re developing your own judgement. Publishing 2 or 3 more articles during the PhD will not replace having a personal judgement that is developed by reading and learning. It’s optimal for the thesis director (lots of article, yay!!!) but it produces PhDs of lower quality in the long term. Advisers die, then what?

13 Derek Jones February 20, 2017 at 10:19 am

I have developed a few simple rules for quickly figuring out whether a paper in software engineering is worth reading:
http://shape-of-code.coding-guidelines.com/2016/06/10/finding-the-gold-nugget-papers-in-software-engineering-research/

and, yes, at least 90% of papers get binned very quickly.

But let’s not forget that reading a paper is rather like watching one episode of a soap:
http://shape-of-code.coding-guidelines.com/2017/01/01/understanding-where-one-academic-paper-fits-in-the-plot-line/

14 Axa February 20, 2017 at 12:44 pm

“After a week or two you should be up to speed on what is happening on the soap you are following.”

Very well put, it takes a while to understand a complex narrative by reading small chapters called articles.

15 carlospln February 20, 2017 at 4:13 am

“If I can find out whether or not the paper is true”

The Econ papers I’ve seen linked to here are hopelessly ridiculous: unrealistic, contrived scope; tricked up ‘mathiness’; shaky conclusions projected back upon the ‘real world’.

None of which make ANY difference to the last three words in the sentence above.

“If I can found out whether or not the paper is true”

You’ve got a lot bigger problems than that, TC.

16 dearieme February 20, 2017 at 4:43 am

If the Thought Police haven’t yet removed this youtube you might find it relevant to this topic.
https://www.youtube.com/watch?v=0sNqnAZTy4w

17 carlospln February 20, 2017 at 6:36 am

nope [thanks anyway]

18 Joe In Morgantown February 20, 2017 at 9:02 am

It’s gone now.

19 JMCSF February 20, 2017 at 6:59 am

What is the value of the bottom third of academic research papers? I even wonder if the top papers are that effective. Can there be a better format than the research paper industrial complex?

I wonder if some professors would be better off spending all their time teaching and mentoring students. Research and teaching are different skill sets, it’s ok to not do both.

I honestly dont know how many people read my brother’s PHD dissertation. I read the first paragraph of his abstract and was like, “honestly who cares?”

20 rayward February 20, 2017 at 7:05 am

Of course, the internet greatly facilitates the ability to push a particular concept or theory or idea, and at the same time provides ready access to far more concepts and theories and ideas. For me and many other readers of this blog, Cowen is our filter, both from the ideas he promotes with his blog posts and his broad reading list he shares with readers. I rely on Cowen because he is respected by his peers, peers with differing views. But I’m neither an ideologue nor credulous. God gave us a brain to think, but it never ceases to amaze me how many don’t bother to use it. The brain is like any muscle: the more one uses it, the better it works.

21 Li Zhi February 20, 2017 at 10:34 am

Some of us here find TC (and AT in a nearly orthogonal fashion) an interesting purveyor of flotsam and jetsam but very very far from a reliable filter. I find few to none of Cowen’s conclusions (or, for that matter his expressed thinking process) valid.

22 Li Zhi February 20, 2017 at 10:40 am

And I have no opinion about whether or not he is respected by his peers. I suppose it partly depends on the definition of “peer”. Based on the quality of most of his posts, I find him very rational, quite reasonable, but rarely astute. No offense. I generally find his analysis lacking in rigor – but note I’m talking about this blog, not his academic (published) work. Given that such a large fraction of what is published “by his peers” in unreplicable garbage, having the respect of the “Social Science” community is hardly something to brag about. Does “respected by peers” and “it’s an old-boys club” mean the same thing?

23 dan1111 February 20, 2017 at 10:53 am

You must be one of those people that thinks it’s ok to say anything, as long as you put “no offense” in there somewhere.

24 albatross February 20, 2017 at 1:40 pm

Published academic articles are very different from blog posts–it would not be reasonable to expect anything like the depth of thought behind a research paper in a blog post.

25 JWatts February 20, 2017 at 2:41 pm

” But I’m neither an ideologue nor credulous.”

Your posting history generally seems to have a strong ideological bias. So, you are ideological, even if you don’t desire to be ideological.

26 Bill February 20, 2017 at 10:08 am

Actually, research sites which indicate the number of downloads and whether the paper was frequently cited actually probably improves the screen of papers than simply calling a colleague and asking for an opinion.

There are screening devices today which tell you not only the citation history but also the network in which the research collaborates in, which also is a screener.

A mathematician neighbor and academic recommended to me a book entitled:

“Why Information Grows: The Evolution of Order From Atoms to Economics” by Cesar Hidalgo.

Great book. Deals with this topic, diffusion, collaboration across fields and innovation, complexity, economic growth, social networks, R and D networks, etc.

27 Li Zhi February 20, 2017 at 10:51 am

Here again (as is reliably true) TC fails to define what he means by researcher. Apparently he wants to speak as a senior member of the science community, but at best – imho – he speaks as a typical product of the academic social science factory. I think it’s fair to say that our understanding of the social sciences, say, economics, has improved enormously in the last 30-50 years. One of the things which has become clear is that the inputs that MUST be addressed are from a far larger swath of human experience compared to what economists (say) were studying 30 years ago. This seems to require a broader reading – with the same (or better) analysis of the “research”. Arguing that the reverse is true, that fewer papers should be scrutinized, sounds to me like mental calcification and a brain which has long ago left the “innovative new thinker” zone.

28 bmcburney February 20, 2017 at 11:06 am

I am not at all sure that the social sciences in general or economics in particular have “improved enormously”, or at all, in the past 50 years. What evidence supports that conclusion?

And why is that “the inputs that MUST be addressed are from a larger swath of human experience.” What “inputs” are we talking about anyway? None of this is defined. You sound just like typical product of the academic grievance industry.

29 albatross February 20, 2017 at 1:43 pm

How would I measure progress in social sciences? Maybe looking for places where their recommendations had been followed and led to good outcomes? The last few decades’ price stability probably suggests something good about macro as applied to monetary policy; the work on designing markets seems to have worked out well in matching med students to residencies, etc. How else would we even know, especially for macro papers that don’t seem susceptible to empirical falsification?

30 bmcburney February 20, 2017 at 3:38 pm

“The last few decades’ price stability probably suggests something good about macro as applied to monetary policy.”

Assuming that improved monetary policy adequately accounts for any improvements in price stability (questionable give policy and outcomes in the very recent past), we should probably ask whether the improvement in that one particular metric is really attributable to “good macro” developed in the past 50 years rather than a policy overreaction to the failure (in terms of price stability) of the previous regime of “good macro” (now the “bad macro” we have more recently overcome). In any case, I believe a connection between price stability and monetary policy was first suggested much more than 50 years ago but even if you count only the “popularization” of those ideas among policy makers we are pushing the envelope to say that this is something which happened in the last 50 years and Li Zhi would be flat out wrong to say it happened in the last 30 years (Milton Friedman won the Noble Prize in 1976).

And assuming that market require that we design them, have we really done better designing markets than Adam Smith would have under similar circumstances?

31 JWatts February 20, 2017 at 2:59 pm

“Arguing that the reverse is true, that fewer papers should be scrutinized, sounds to me like mental calcification and a brain which has long ago left the “innovative new thinker” zone.”

I would agree that I found Tyler’s advice to be rather surprising.

“Tyler: They do/should spend more time reading the papers where they are the expert evaluators, but that may mean reading fewer papers overall.”

This would seem to lead to likely Silo-ization, which seems particularly odd coming from Tyler, who seems to pride himself on reading from a wide variety of topics.

32 Turkey Vulture February 20, 2017 at 4:21 pm

It is probably good advice for most people who might want to “advance knowledge,” as most of the advancement will take place within those silos and involve small increments.

But for a minority of people who might tie disparate areas together in new ways, it would be bad advice.

It’s tough to give the advice with that caveat, though, since most people will want to think they are going to be the ones who come up with something new and clever.

33 bmcburney February 20, 2017 at 10:56 am

“the mistakes become less and less transparent.”

Indeed, and, of course, “mistakes” are the least of it. Intentional faking of results also becomes less transparent and the spurious results, whether intentional or not, can build on and from each other just like real results.

34 gw February 20, 2017 at 11:32 am

I would subscribe to a podcast that gave 10-15 min summaries of recent papers. Especially if it was hosted by someone I trust.

35 albatross February 20, 2017 at 1:45 pm

TWIV and TWIM are microbiology podcasts that usually cover 2-3 papers in an hour or so, sort of like getting to listen in on a journal club meeting, but with more asides to explain unfamiliar terms and ideas.

36 richard February 20, 2017 at 12:10 pm

There is quite a bit of difference between how and why a paper is read by a professional research scientist (in the information-dense natural sciences at least), and the general commentary on how papers are evaluated, whether the findings are ‘false’ or not, etc.

The results of any given paper are generally just not that relevant. Most of the time that I read a paper, I’m just looking for some bit of information that may or may not be important to the major research question or results of the paper. I want some data, or an equation, or I just need a paper to cite so that I don’t have to derive something from first principles. Furthermore, both data and theory have a shelf life, and after a few years one or both may be obsolete (with data it’s because we are always getting better at measuring things, and time series just keep getting longer too).

The same goes for the papers that cite my work, or cite the papers that I cite. A real, critical engagement of the research question or results is pretty rare. Usually people just want some small nugget of information for their own purposes.

I think this is an outcome of how the networks of researchers and ideas are constructed, and is very natural and not generally a problem. Given a wide range of interests out there, it’s far more likely for my work to be only peripherally relevant to some topic than for my work to be central to it. I have many acquaintances, but only one wife.

Additionally, as more and more papers are coming online (a lot of high-quality papers from before the 90s have only been digitized in the last few years), more papers are browsed and fewer are read thoroughly, because discovery and browsing are easier, and reading is harder (hard copies are better for notes and less distracting). This has changed how I write papers as well–I put less work into making a paper that flows nicely and more into one that is more easily browsed by the peripheral or incidental reader. I also try to make the papers as short as possible, and to make all data, equations and code very accessible. Graduate students are the only people who consistently read papers from start to finish, or who really care what some random author thinks about some process.

There are some exceptions–papers by the luminaries in the field, more theoretical papers, or papers that are the first to identify and properly characterize some phenomenon. These latter ones tend to be comprehensive in a way that follow-up papers are not, and a lot of insight can be gained by following the line of reasoning from observations to characterization and validation of some idea.

37 Troll me February 20, 2017 at 7:12 pm

Could be connected to the question of whether administrative staff in universities has exploded in a manner which is reasonable, or has exploded in a manner which is unreasoanble.

Profs spend less time dealing with admin stuff now, right? So … if they are also reading fewer papers, what are they doing with their time? Blogging? Exchanging Facebook and Twitter feeds?

38 Hazel Meade February 20, 2017 at 11:02 pm

The difficulty then, is figuring out who should be reviewing what papers, without accidentally establishing a dogma through some feedback loop in which the people selected to review the papers are biased to reject ideas not part of the current dogma. .

39 Troll me February 22, 2017 at 1:05 am

I think on average it’s usually sufficient to count on people wanting to produce and publish “the next big thing”. This is a major source of contrarianism which may lead to important discoveries, or at worst shaking it up a bit.

Within a given publication, department, institution, etc., there could be issues. But with many publciations and institutions always trying to get ahead, there is lots of incentive to continue with exploration.

It helps when disagreeing with the president or partisan perspective is not a career-ending move. (I’m not talking about race and gender here, but those seem to be the most important and most relevant areas of research to a number of participants on this board. Perhaps … it could be possible to talk about education-related stuff without a focus on gender and race?)

40 Kevin James February 21, 2017 at 6:56 am

There is (as always) a literature on this topic. A “reliance on authority” screen will surely strengthen the “science advances one funeral at a time” effect (http://www.nber.org/digest/mar16/w21788.html).

41 Jackson Layers February 21, 2017 at 10:00 am

It is vital that whatever we do, we do it with proper research instead of anything else. I do Forex trading and it is easily the best work especially with broker like OctaFX, as they are special with excellent educational guidance plus there is also demo contest like cTrader available, so I can participate there and get great experience which really helps me out with working while the prize which is up to 400 USD makes it up for excitement.

Comments on this entry are closed.

Previous post:

Next post: