Regression Discontinuity Works

Robert Lalonde’s famous 1986 paper, Evaluating the Econometric Evaluations of Training Programs with Experimental Data, shattered the confidence of the profession by showing that the advanced econometric techniques of the day, by and large, failed to recover the results from a randomized controlled trial. The profession has been busy since that time developing new methods and techniques.

A new paper compares regression discontinuity with RCTs and RD works very well.

Theory predicts that regression discontinuity (RD) provides valid causal inference at the cutoff score that determines treatment assignment. One purpose of this paper is to test RD’s internal validity across 15 studies. Each of them assesses the correspondence between causal estimates from an RD study and a randomized control trial (RCT) when the estimates are made at the same cutoff point where they should not differ asymptotically. However, statistical error, imperfect design implementation, and a plethora of different possible analysis options, mean that they might nonetheless differ. We test whether they do, assuming that the bias potential is greater with RDs than RCTs. A second purpose of this paper is to investigate the external validity of RD by exploring how the size of the bias estimates varies across the 15 studies, for they differ in their settings, interventions, analyses, and implementation details. Both Bayesian and frequentist meta‐analysis methods show that the RD bias is below 0.01 standard deviations on average, indicating RD’s high internal validity. When the study‐specific estimates are shrunken to capitalize on the information the other studies provide, all the RD causal estimates fall within 0.07 standard deviations of their RCT counterparts, now indicating high external validity. With unshrunken estimates, the mean RD bias is still essentially zero, but the distribution of RD bias estimates is less tight, especially with smaller samples and when parametric RD analyses are used.

Comments

What results? The pioneer in p-hacking is a Greek math prodigy, Ioannidis.

?

I assume you mean the pioneer in criticising p-hacking. Or something like that. Your comment was rather ambiguous.

I'm a bit more skeptical of RD based on historical data. We can't do time travel RCT and a lot depends on being able to identify all the possible confounds and correcting for selection bias. Lots of current work doesn't even adjust for human capital, biology, personality, cultural predisposition, or genes and just waves its hands about this. But this is the persistence RD that is hot in the development literature.

The whole point of RD is that you don't need to identify confounds.

You do if there is selection in which groups persist across boundaries over time and space. That's the point about historical persistence.

Wiki, you are incorrect. Do read the links, RD is not regression with any type of discontinuity.

It's fine if these things persist across the boundaries. The problem is when those variables that make groups distinct are correlated or concomitant with the cutoff used in the RD design. So whether or not researchers are controlling for "human capital, biology, personality, cultural predisposition, or genes", researchers can still get causal estimates of the mechanism under study as long as those "confounders" are not correlated with the RD cutoffs. If you have a concern that something is not being controlled for, you have to make the case that those factors differ systematically with the RD boundaries.

Yawn.

There are two kinds of dents economic studies:

1. Those that confirm my biases and thus are worthy contributions.

2. Other econometric studies, which have methodological flaws.

Comments for this post are closed