Are IVs Going the Way of the Atlantic Cod?

by on February 2, 2011 at 7:05 am in Economics, Science | Permalink

It's hard to come up with a good instrumental variable (plausible source of exogenous randomization) so when someone does come up with one (e.g. legal origin) it's tempting to want to use it again and again. Unfortunately, as Randall Morck and Bernard Yeung point out in Economics, History and Causation, IVs with more than one use are deeply problematic. If a variable is a good IV for X then it can't also be a good IV for Y without also controlling for X and vice-versa. What this means is that every new use of an IV casts doubt on every previous use. Or as, Morck and Yeung, memorably write:

A Tragedy of the Commons has led to an overuse of instrumental variables and a depletion of the actual stock of valid instruments for all econometricians. Each time an instrumental variable is shown to work in one study, that result automatically generates a latent variable problem in every other study that has used or will use the same instrumental variable, or another correlated with it, in a similar context. We see no solution to this. Useful instrumental variables are, we fear, going the way of the Atlantic cod.

I am not quite so pessimistic, I don't see this as a fundamentally new problem or one specific to IVs. Nevertheless, I appreciate their advocacy for a wide variety of empirical methods as a solution to the over-fishing problem.

Among the alternative methods Morck and Yeung recommend, are event studies and Granger causality. I couldn't help but laugh at the last. But I wholeheartedly applaud their primary recommendation which is greater use of and respect for narrative history.

1 steve February 2, 2011 at 3:09 am

Agreed, AT. Every time someone makes that suggestion it Granger causes me to crap myself.

2 Andrew February 2, 2011 at 3:32 am

One point for praxeology? Now I'm talking way out of my asset class.

3 Don Taylor February 2, 2011 at 3:43 am

Where did Atlantic Cod go? Also, thinking is useful when considering IVs.

4 Gabe Harris February 2, 2011 at 4:30 am

I hope this article is toungue in cheek. It isn't the new studies that devalue the old conclusions….it is the application of logic in the first place. This is what the Austrians have been saying for years!

5 Rahul February 2, 2011 at 4:43 am

Ignorance is bliss?

6 a meanie February 2, 2011 at 6:00 am

non-economist,

It means 20th century economics was not 'science', but a load of self-referential path dependent social engineering organized by the Rockefeller Foundation who literally invented "social science" at the University of Chicago.

7 Pierre-Louis February 2, 2011 at 6:20 am

"If a variable is a good IV for X then it can't also be a good IV for Y without also controlling for X and vice-versa."

this is not true if the dependant variable in the second regression is not correlated with X. Then the IV can be good for Y without controlling for X.

8 anon February 2, 2011 at 6:34 am

I agree with "Justa Commenter" above.

I think the problem existed before the IV was reused.

Framing this as a commons problem is highly misleading. The correct answer would be to question the validity of the first study in the first place.

9 Jeju February 2, 2011 at 6:38 am

Great post! More of that, please!

10 anonymous February 2, 2011 at 6:50 am

This is a great post, and I just printed out the article which also looks great. I don't understand all the points made in Alex's post – I'm going to have to write out the algebra to probably understand what is otherwise an easy point to grasp because I'm stupid – but that there are diminishing returns to any instrument is something I've thought about for a while.

One thing that is also different is that as our understanding of the IV estimator improves, the total stock of instruments at any point in time falls as well. Case in point is the weak instrument literature – many of the really famous IV papers by people like Hoxby, Card, Levitt or Angrist predate the weak instrument literature (Angrist plays a role in the creation of that literature with his birth quarter paper with Krueger). Now suddenly it's not enough to have plausibly exogenous variation; we must also have hyper strong correlations between the instrument and the endogenous variable. Add to that the growing econometric literature on serial correlation in panel models and "clustering", and you end up with even fewer instruments that can pass all the statistical diagnostics.

I think too that seeing some of these earlier IV papers successfully challenged for a variety of reasons – Hoxby's rivers and streams instrument, Levitt's police hirings instrument, Angrist's birth quarter instrument – combined with what feels like a surplus of papers that used IV since the early 2000s has possibly made editors and referees harder to please. "Cute instruments" has become a pejorative in applied work, whereas it was once a complement to say that about someone's paper.

Plus, the rise in field experimentation in development, and increasingly in labor and health economics, may be decreasing the value of IV's marginal product. My confidence parameters on my own IV papers always feel wider than what is reported, for instance. I just can never fully believe that what I did really was valid, and part of that is because the excludability of the instrument is an assumption in 2SLS that is not testable. So how do I really know my instrument is valid? I can believe it, and I can try to convince other people, but I know that deep down there's a bias incentive in my deductions and arguments because the only way my great results can work is if the assumption holds. And when you add to that already the other things, some people just have no incentive to believe that assumption – or if they are deep at the point of diminishing returns to IV, they may be biased against not believing it. Consider field experimentation – the econometrics are simple. This is just comparison in means between the randomly assigned treatment and control groups. When we can actually test directly many of the things Hoxby and others were doing indirectly with IV, how can it not make IV less valuable at the margin? IV has always been a second best solution in the social sciences created by a world of non-experimental data generating processes. But it's trying to approximate the experimental data generating process if all the conditions hold (hence "natural experiment" nomenclature). So if we can actually do the experiments, then of course we are going to prefer that.

11 Luis Enrique February 2, 2011 at 7:08 am

sounds similar to the argument made in this paper: Blunt Instruments

12 Sam February 2, 2011 at 7:09 am

In a Center for Global Development Working Paper, Michael Clemens and I made a similar argument, among others, in the context of understanding the determinants of economic growth: http://www.cgdev.org/content/publications/detail/

Self-promotion notwithstanding, I agree with Morck and Young that this issue goes underappreciated by most empiricists.

13 Stuart Buck February 2, 2011 at 7:21 am

Within the space of 1.5 minutes, three separate people posted about the "Blunt Instruments" paper! Off to read it . . . .

14 Bernard Yomtov February 2, 2011 at 7:55 am

justa commenter's point seems sensible to me. I don't see why the problem wasn't always there, even before the first use of the IV.

It's entirely possible I'm missing something.

15 Daniel February 2, 2011 at 9:34 am

Analogy has me puzzled too – what aspect of this recalls a tragedy of the commons?

"If only damn empiricists would stop reusing IVs we could stay blissfully ignorant of the bias in our results to date!"(?)

If the original IV is untenable with subsequent results – then overfish! Throw doubt on the papers.

16 Mr, Ed February 2, 2011 at 1:40 pm

after medical researchers reinventing integration
http://fliptomato.wordpress.com/2007/03/19/medica

economists reinvent the rank condition in systems of simultaneous equations?

nothing to see here people, move on…

17 RJB February 3, 2011 at 3:26 am

I don't have time to read the article to see whether the mistake is Alex's or the authors, but Alex's argument has several flaws. These have been alluded to above, but the guiding force of <a href = "http://www.ftrain.com/wwic.html"&gt; WWIC compels me to write.

1. The only determinant of the utility of an instrumental variable is its correlation with the dependent, independent and potentially relevant omitted variables of interest.

2. The fact that someone has written a paper using X as an instrumental variable can only be helpful to someone else who wants to do so, because it provides more insight into the behavior of X and its correlations with other variables.

3. Problems in interpreting X as a source of exogenous variation may be more apparent as we gain knowledge about X, and may cause more skepticism by readers (peer-reviewers!) and more work by the author, but does not actually alter any correlations, and hence does not alter the validity of the research approach.

As a result,

4. The only "resource" lost to this tragedy of the commons is the authors' ability to persuade people with little effort that an inappropriate use of an instrument is actually appropriate.

18 stickman February 3, 2011 at 5:29 am

I'm with those stating the "tragedy of the commons" analogy is a poor one. The requirements for IV hold whether a particular instrument has been used before or not… Indeed, prior use adds to our validity testing (as RJB has succinctly laid out above).

Comments on this entry are closed.

Previous post:

Next post: