Month: May 2017

Paul Krugman blogged on that, with initial impetus from Noah Smith.  Here is Noah:

If you and your buddies have a political argument, a vast literature can help you defend your argument even if it’s filled with vague theory, sloppy bad empirics, arguments from authority, and other crap. If someone smart comes along and tries to tell you you’re wrong about something, just demand huffily that she go read the vast literature before she presumes to get involved in the debate. Chances are she’ll just quit the argument and go home, unwilling to pay the effort cost of wading through dozens of crappy papers. And if she persists in the argument without reading the vast literature, you can just denounce her as uninformed and willfully ignorant. Even if she does decide to pay the cost and read the crappy vast literature, you have extra time to make your arguments while she’s so occupied. And you can also bog her down in arguments over the minute details of this or that crappy paper while you continue to advance your overall thesis to the masses.

…My solution to this problem is what I call the Two Paper Rule. If you want me to read the vast literature, cite me two papers that are exemplars and paragons of that literature. Foundational papers, key recent innovations – whatever you like (but no review papers or summaries). Just two. I will read them.

If these two papers are full of mistakes and bad reasoning, I will feel free to skip the rest of the vast literature. Because if that’s the best you can do, I’ve seen enough.

Those are both interesting posts, but my perspective is different, probably more as a matter of temperament than thinking they are objectively wrong.  Here are a few comments:

1. The best two papers on ethics are not very convincing.  Nonetheless people who have worked their way through a good amount of that literature are much better at ethical reasoning than those who have not.

2. The best two papers on global warming are not very convincing.  What is convincing is how many different perspectives and how many different branches of science point toward broadly similar conclusions.  In fact the aggregate effect here is quite overwhelming (don’t debate gw in the comments, not today; I’ll delete).  It is a question of many moats, not all of them being entirely muddy.

3. I see the Smith-Krugman standard as fairly economistic, and fairly MIT-late 20th century at that.  It is one vision of what a good literature looks like, and a fairly narrow one.  It will elevate simple answers in status, whether or not that is deserved.  It discriminates against dialogic knowledge, book-based knowledge, historical knowledge, and knowledge when the answers and methods are not very exact.  There is the risk of ending up too certain about one’s knowledge.

That all said, I do understand that specialized top researchers, including Nobel Laureates, often may do better holding relatively narrow methodological visions.  Look at all the Nobel Prizes that have been awarded to Chicago.  It might be entirely correct to insist that Becker’s treatise on the family pay more attention to anthropology, but that doesn’t mean he should have followed that advicee.

4. The standard seems to discourage reading, and I would not want to teach it to my students.  I teach something more like “always read more, unless you are writing or doing relevant quantitative work.  And one reason you write is to improve the quality of your reading.  Read more and write more, all the time.”  I still think that is better advice for most (not all) people.

5. Isn’t there a lot to be said for deferring to the opinions of those who have read through the “muddy moat”?  By no means are they all partisans, and the non-partisan ones care most of all about the truth.  After all, they did all that reading!  Defer, rather than trust so much in your ability to pick you the right two papers, or have someone pick them out for you.  I have a much more positive view of survey articles than does Noah, while understanding they do often leave you fairly agnostic on major issues.

6. If the truth of the matter is in fact muddy, you may need to dip into the muddy moat to learn that.

7. The difference between total value and marginal value may be relevant.  You might conclude a field literature has low total value, but the marginal value of learning more about that area still could be quite high.  That is in part because muddy fields and results don’t spread so readily, and so dipping into the muck can yield some revelations.  That is another reason why I would not offer the “two paper standard” as practical advice.

8. If anything, I would put the reading pressure on the other side, namely more rather than less.  Rather than encouraging readers to dismiss or downgrade fields, I would urge them to consult different disciplines altogether, including political science, sociology, and anthropology, others too.  This is much easier to do if you take a more positive attitude toward survey articles.

9. This is quite a subjective impression, but I worry that the dogmatic will use the two paper standard to dismiss or downgrade particular lines of investigation.

10. I don’t know if Noah and Paul were referring to my colleague Garett Jones, who frequently tweets “…if only there were a vast empirical literature” when he sees claims that he regards as empirically false.  Now, I am not the Garett Jones oracle, but I always took his use of the word “vast” to be slightly sarcastic.  Usually these are cases where even a fairly cursory knowledge of the literature in question would indicate something is wrong with the claim at hand.  In my view, Garett is not demanding “vastness” of effort, rather he is criticizing those who don’t grasp what the effort space looks like in the first place.