The market for lemmas

It’s not that big:

Of the sixty articles in AER, EJ, JPE, and QJE that had been cited more than 500 times, only one article contained an author written lemma.

Here is the paper, and thanks to Johan Almenberg for the pointer.  On p.6 you see a good graph, "Lemma frequency by decade."  Note also that a top paper in Econometrica is much more likely to have a lemma.

Comments

Lemmas are for lemmings.

Well yeah, but according to your link there were only ca 17 journal articles per year with Lemmas 1970-1990, the period most of the cited papers are from (35 per year 1990-2000).

How many articles were published per year in those journals? 4-500? So you have 3-4% of articles with lemmas, and 1,7% of the top 60 citations? The under representation may be statistically significant, but not very impressive, especially in such a small sample. One more would have made Lemmas more likely to be cited†¦

Anyway I am not sure what the point is, except the obvious fact that there is a life cycle with important new theories. The first idea is more likely to be expressed verbally, subsequent authors write more rigorous versions.

Unless you are Stigler or Kahneman there is not much this tells you.

Seems to be more proof that theoretical researchers don't talk to academic empirical researchers (and do either dialogue much with practitioners?)

So, if a lemma never gets empirically tested, does it make a sound?

Why in the world do people ever double space their papers? It always makes me think of freshman term papers. How painful.

How about this: Papers without lemmas are accessible by many, many people, and papers with lemmas are accessible by a much narrower set of people. Akerlof's 1970 QJE on lemons markets is much more accessible to the layscholar than Charles Wilson's 1980 Bell Journal on equilibrium in markets with adverse selection. So the layscholars skip Wilson's piece because they can't/don't want to slog through it, but cite Akerlof's piece because it's an easier read.

In the econometrics literature, I wonder how many people have actually READ Hausman's 1978 Econometrica on specification tests, as opposed to simply citing it because "it's that specification test paper".

jim,

Thanks for the reference. As, I said I'm inclined to believe the result. But I'll stand by my claim that the "lemmas" paper shouldn't shift one's priors much, and to be honest, while the analysis in the SEJ paper is slightly better, it isn't exactly great either.

The SEJ paper claims to find two things:

(a) Papers with lemmas (PWLs) are themselves less likely to have "empirical content" than papers without lemmas. This seems entirely unsurprising. It's consistent with the Gordon hypothesis, but only in a way that makes it almost trivially true: complexity crowds out empirics because, by definition, you've spent your time generating lemmas rather than analysing data.

(b) PWLs have a lower proportion of citing articles with empirical content. This seems closer to what we want, but the paper unfortunately doesn't define it's comparison group adequately. PWLs could generate a smaller proportion of empirical papers either: (i) because the Gordon hypothesis is true and more complex theory papers generate fewer empirical citations than simpler theory papers; or (ii) simply because (per (a)) PWLs are more likely to be be theory papers rather than purely empirical papers, and theory papers tend to generate a higher proportion of theory papers than do purely empirical papers. It should have been pretty easy to rule out the second explanation, but they didn't try to.

(As an aside, I also suspect that the authors increase the significance of their results by artificially inflating the sample size (they count citing articles rather than cited articles) but I'm not sure that would affect the result much.)

jim, i don't think i'm quibbling with nuances. i'm saying that the studies don't adequately test the hypotheses they say they do. consequently, they fail to refute the null that the Gordon hypothesis is false. by your reasoning, this would seem to imply that "as the anti-Gordon hypothesis fails to be refuted by test after alternative test its merits are becoming more and more clear."

to put it more bluntly: a bunch of crap studies don't necessarily add up to convincing evidence. they may just add up to a bunch of crap.

i find this particularly annoying because, as i said, i'm a priori inclined to believe the hypothesis, and some really simple checks in both papers should have fixed the problems i've pointed out. when people don't bother to report really obvious checks like this, there are two potential reasons: (i) they didn't think to run some really obvious tests; or (ii) they actually ran the checks, but didn't the results they wanted. neither of these possibilities should exactly incline one to revise one's priors upwards.

if you want to argue that the Gordon hypothesis is probably true, i'll concur. if you want to argue that these studies contribute anything substantive to proving that, you'll find serious disagreement.

Jim, Explanations in science win out because they do a better job of predicting the evidence than alternative explanations. The problem with these papers is that they present evidence that one would place a high prior probability on observing even if Gordon's hypothesis is false. They consequently provide little reason for updating our beliefs about the hypothesis.

Feel free to keep handwaving about how this is a mere quibble, peer review is infallible, and that the fact that Gordon's hypothesis hasn't been contradicted means that it must be true. Repeating it won't make it any more correct. I think I've wasted more than enough time on this already, particularly given that you seem unwilling to engage with substantive argument.

indiana jim,

Why is it important that economic theories must lead to "operational statements" that can be tested? Haven't you read Rubinstein's 2006 Econometrica paper (http://arielrubinstein.tau.ac.il/papers/74.pdf)?

Hi Best wishes。Allow me to offer my heartiest wishes.xicao loves-流水线娱乐博客常年提供高、中、低压锅炉钢管、流体钢管、结构钢管、化肥专用钢管、石油裂化钢管、地质钢管、液压支柱钢管及合金钢管-无缝管-无缝钢管等论文发表资讯/刊物信息,协助客户制定论文发表方案

Comments for this post are closed