The Roodman Replication

David Roodman, working for the Open Philanthropy Project, has completed an absolutely tremendous replication and extension of many papers in the literature on deterrence and crime. He reaches two conclusions. First:

I estimate, that at typical policy margins in the United States today, decarceration has zero net impact on crime. That estimate is uncertain, but at least as much evidence suggests that decarceration reduces crime as increases it. The crux of the matter is that tougher sentences hardly deter crime, and that while imprisoning people temporarily stops them from committing crime outside prison walls, it also tends to increase their criminality after release. As a result, “tough-on-crime” initiatives can reduce crime in the short run but cause offsetting harm in the long run.


Empirical social science research—or at least non-experimental social science research—should not be taken at face value. Among three dozen studies I reviewed, I obtained or reconstructed the data and code for eight. Replication and reanalysis revealed significant methodological concerns in seven and led to major reinterpretations of four. These studies endured much tougher scrutiny from me than they did from peer reviewers in order to make it into academic journals. Yet given the stakes in lives and dollars, the added scrutiny was worth it. So from the point of view of decision makers who rely on academic research, today’s peer review processes fall well short of the optimal.

My paper on Three Strikes with Eric Helland was one of the papers that Roodman replicated. (Fortunately, it did replicate with the exception of one error in a table.) I can vouch that Roodman gave us tougher scrutiny than did the peer reviewers.

Not surprisingly, I don’t agree with all of Roodman’s conclusions but rather than pushing back I think it more important to underline how impressive the replication project is. There are many review papers in economics but a replication project of this magnitude is nearly unprecedented. In our paper on the National Science Foundation, Tyler and I advised the NSF to put more efforts into replication. We wrote:

The NSF could support replication studies on a significant scale. A significant fraction of economic research does not easily replicate…Replication and reproducibility studies are true public goods that are not rewarded highly by most top journals or by the tenure process at research universities.

Roodman and OPP have demonstrated the value of replication on a large scale.


Comments for this post are closed