The Roodman Replication

by on September 26, 2017 at 7:25 am in Economics | Permalink

David Roodman, working for the Open Philanthropy Project, has completed an absolutely tremendous replication and extension of many papers in the literature on deterrence and crime. He reaches two conclusions. First:

I estimate, that at typical policy margins in the United States today, decarceration has zero net impact on crime. That estimate is uncertain, but at least as much evidence suggests that decarceration reduces crime as increases it. The crux of the matter is that tougher sentences hardly deter crime, and that while imprisoning people temporarily stops them from committing crime outside prison walls, it also tends to increase their criminality after release. As a result, “tough-on-crime” initiatives can reduce crime in the short run but cause offsetting harm in the long run.

Second:

Empirical social science research—or at least non-experimental social science research—should not be taken at face value. Among three dozen studies I reviewed, I obtained or reconstructed the data and code for eight. Replication and reanalysis revealed significant methodological concerns in seven and led to major reinterpretations of four. These studies endured much tougher scrutiny from me than they did from peer reviewers in order to make it into academic journals. Yet given the stakes in lives and dollars, the added scrutiny was worth it. So from the point of view of decision makers who rely on academic research, today’s peer review processes fall well short of the optimal.

My paper on Three Strikes with Eric Helland was one of the papers that Roodman replicated. (Fortunately, it did replicate with the exception of one error in a table.) I can vouch that Roodman gave us tougher scrutiny than did the peer reviewers.

Not surprisingly, I don’t agree with all of Roodman’s conclusions but rather than pushing back I think it more important to underline how impressive the replication project is. There are many review papers in economics but a replication project of this magnitude is nearly unprecedented. In our paper on the National Science Foundation, Tyler and I advised the NSF to put more efforts into replication. We wrote:

The NSF could support replication studies on a significant scale. A significant fraction of economic research does not easily replicate…Replication and reproducibility studies are true public goods that are not rewarded highly by most top journals or by the tenure process at research universities.

Roodman and OPP have demonstrated the value of replication on a large scale.

1 rayward September 26, 2017 at 7:40 am

I realize that Tabarrok’s post is about replication and not incarceration per se, but I have to add my two cents about mandatory minimum sentences. I was working for my state’s legislature (specifically the judiciary committee) when mandatory minimum sentences became the end-all solution to crime. Part of my job in writing the committee report was to estimate the cost of the mandatory minimum sentence, which I did by taking the difference in the minimum sentence and the average sentence served for the particular crime and multiplying it by the average daily cost of incarceration. The committee members strongly objected to my approach, believing as they did that the minimum sentence would be such a deterrent to crime that the number incarcerated would fall and the total cost to the state would plummet. That was back in the mid-1970s. Since, the number incarcerated, the number of prisons, and the total cost to the state exploded. And the legislature’s response has been to add more mandatory minimum sentences, believing that the minimum sentence would be such a deterrent to crime that . . . . groundhog day! Now that’s replication.

Reply

2 rayward September 26, 2017 at 8:55 am

The mandatory minimum sentence is but an early iteration of the free lunch approach to policy. Today the free lunch approach is reflected in “dynamic scoring” and the belief that tax cuts, especially those at the upper end of the income scale, will more than pay for themselves with a surge in economic growth. Who should be blamed for this nonsense: the politicians who promote it, the economists who provide them with cover, or Americans who fall for it?

Reply

3 Pshrnk September 26, 2017 at 11:22 am

@rayward ” Who should be blamed for this nonsense: the politicians who promote it, the economists who provide them with cover, or Americans who fall for it?”

All The Above

Reply

4 Anonymous September 26, 2017 at 3:12 pm

None of the above because no one has that belief or argues for it

Reply

5 So Much For Subtlety September 26, 2017 at 8:55 am

New York jailed so many people that crime collapsed. Now prisons are being closed. New York is saving a great deal of money not only from lower crime rates (what is the value of a girl not being raped by the way?) but also from lower costs of incarceration.

Your home town clearly did not jail enough people.

Reply

6 Art Deco September 26, 2017 at 9:05 am

New York invested more in increasing the police census and modifying police deployments and tactics. However, New York also trebled the prison census as well. New York’s experience suggests that you could rebalance between police spending and prison spending to improve your results (which is a point distinct from one suggesting that imprisoning people has no effect on the frequency of crime. If it was the moderator’s point that we’d reached a plateau at some point in the marginal contribution of adding to the prison census, he could have said that).

Reply

7 MOFO September 26, 2017 at 9:12 am

Its always worth noting that New York has most likely been juicing its crime stats for some time:

http://www.huffingtonpost.com/2013/07/12/nypd-compstat_n_3587637.html

” In 2010, they released the results of a survey in which dozens of retired police officials complained that pressure from department brass prompted widespread statistical manipulation of CompStat data, specifically by downgrading reports of serious crimes to less serious offenses.

The outside audit released last week not only confirmed that such data manipulation takes place but found several weak points in the ways the department tracks and uncovers it.”

http://www.nytimes.com/2010/02/07/nyregion/07crime.html

“More than a hundred retired New York Police Department captains and higher-ranking officers said in a survey that the intense pressure to produce annual crime reductions led some supervisors and precinct commanders to manipulate crime statistics, according to two criminologists studying the department.”

“Others also said that precinct commanders or aides they dispatched sometimes went to crime scenes to persuade victims not to file complaints or to urge them to change their accounts in ways that could result in the downgrading of offenses to lesser crimes, the researchers said.”

Etc etc. There are more, worth looking at, YMMV and all that.

Reply

8 Art Deco September 26, 2017 at 11:13 am

Its always worth noting that New York has most likely been juicing its crime stats for some time:

No, but people have an emotional investment in thinking that so. I take it you fancy they’re hiding dead bodies. If they were cooking the books, there would be discrepancies between statistics collected through collating police reports and statistics collected through survey research.

Reply

9 MOFO September 26, 2017 at 11:33 am

I dont fancy anything, but i have read a number of sources claiming that the NYPD have in the past been under pressure to reclassify crimes as of lesser importance or pressure victims to not file reports at all to keep crime stats down.

Im no expert in this by any stretch, but im not sure that the crime statistical reports you mention (i assume you are talking about NCVS or similar) are setup to detect discrepancies in the official data. Its possible that one *could* detect statistical shenanigans, but ive seen no proof that anyone actually is. Again, this is not my bailiwick, so if you have better info, id love to see it.

I have no emotional investment either way, but only a fool ignores information to protect his prior held beliefs.

I know this is not about New York, but here is an article about Chicago doing exactly what i described, so its not that far fetched to believe that NY is doing the same thing.

http://www.chicagomag.com/Chicago-Magazine/May-2014/Chicago-crime-rates/

http://www.chicagomag.com/Chicago-Magazine/June-2014/Chicago-crime-statistics/

“But New York’s CompStat program was dogged by talk that its numbers weren’t all that they seemed. In May 2010—four years after McCarthy left the NYPD to become the top cop in Newark, New Jersey—The Village Voice published excerpts of transcripts from hundreds of audio recordings of station house roll calls from 2008 and 2009 made by an officer in the 81st Precinct, in Brooklyn’s tough Bedford-Stuyvesant neighborhood. NYPD bosses were caught on tape pressuring street cops to downgrade crimes and discourage victims from reporting offenses, among other things, to bolster the precinct’s stats. An outside report ordered by former police commissioner Raymond Kelly would confirm that manipulation went well beyond one precinct and “may have an appreciable effect on certain reported crime rates.”

Pervasive? Dont know. Still happening? Also dont know, but its worth noting that cites do have an incentive to lie about their crime numbers, whether they are using a comp stats type system or not.

10 Art Deco September 26, 2017 at 1:29 pm

I dont fancy anything, but i have read a number of sources claiming that the NYPD have in the past been under pressure to reclassify crimes as of lesser importance or pressure victims to not file reports at all to keep crime stats down.

No, you have newspaper articles penned by people hostile to a particular project drawing on vague characterizations from faceless groups (“dozens” in a force that employs over 40,000 people) and paraphrases of what a police supervisor supposedly said 8 years ago.

11 MOFO September 26, 2017 at 1:39 pm

Your response seems to be “i dont like this, so you are wrong”. The articles i linked to werent anonymous, they named who did the studies and what they found. Maybe you disagree with them, but you are not making a very good case that your disagreement is anything more than hand-waving to protect your prior held beliefs. Not everyone who says something you disagree with is necessarily hostile to what you believe, its just that some of us prefer to base our beliefs on actual fact, not table pounding. Normally, you do too, but your unwillingness to do so in this case tells me you really dont have any facts, but would prefer we all look away anyway.

12 KRing September 26, 2017 at 11:32 am

rayward, please reach out to me at kring@famm.org. As someone still fighting lawmakers on the utility of mandatory minimums, especially their deterrent value, I would love to hear more about your experience.

Reply

13 Sam Haysom September 26, 2017 at 11:44 am

Famous last words.

Reply

14 Bill September 26, 2017 at 8:25 am

Tyler’s and Alex’s paper on the NSF linked above is well worth reading. Good work and good recommendations.

Reply

15 Art Deco September 26, 2017 at 8:34 am

And if it all works out very badly, I’m sure you’ll studiously ignore that and respond with evasions when you’re cornered.

Reply

16 Reston September 26, 2017 at 8:48 am

“Replication and reproducibility studies are true public goods”

Nonsense

Regrettably, many “libertarians” readily concede some alleged “public goods´´ to the statists… but then they are on a slippery slope– for there is no limit to the “public goods argument”.

Reply

17 aMichael September 26, 2017 at 12:51 pm

There is a limit in what is actually a public good. It’s limited to things that are non-rivalrous and non-excudable. It turns out that in the strict sense, few things qualify as public goods. Knowledge (which includes the knowledge that a finding replicates) is one of the few ones. (I.e., if I come up with an idea of how to do something better, anyone else can use that idea, too, once they learn of it.)

The issue, that maybe you’re going with, is how do you determine how much of a true public good to provide either through laws (i.e., copyright laws for knowledge) or public spending (i.e., NSF grants). But, yeah, if you think government shouldn’t fund or regular anything, then you’ll have to live with a lower provision of public goods than would be optimal.

Reply

18 JonFraz September 26, 2017 at 1:37 pm

There is a limit in what is actually a public good. It’s limited to things that are non-rivalrous and non-excudable.

Only if you accept that arbitrary definition– which was not handed down from Mt Sinai after all.

Reply

19 Anonymous September 26, 2017 at 3:15 pm

I suppose all dictionary definitions are arbitrary in a sense

Reply

20 So Much For Subtlety September 26, 2017 at 8:52 am

it also tends to increase their criminality after release

Which seems a little contradictory. They say letting people out does not increase crime and yet letting people out causes an increase in crime.

The sensible solution is not to let people out. The problem is not with prisons. The problem is with releasing people. The longer they are kept in, the safer the rest of us are.

Is there any point reading this study? It looks like the usual pro-criminal nonsense we have all seen before. It is this sort of thing that gives academics a bad name.

Reply

21 toby September 26, 2017 at 9:56 am

…. The people we imprison are overwhelmingly our most disadvantaged: the poor & poorly educated, the black & brown, the mentally ill. Typically, they’re given extraordinarily long sentences compared to prisoners in the European Union, often for infractions that would not even warrant incarceration elsewhere.

There are hundreds of thousands of “offenses” for which YOU as an American might be imprisoned.
YOU (and all Americans) have no clue as to the totality of “law” you live under.

Reply

22 Art Deco September 26, 2017 at 11:16 am

. The people we imprison are overwhelmingly our most disadvantaged: the poor & poorly educated, the black & brown, the mentally ill. Typically, they’re given extraordinarily long sentences compared to prisoners in the European Union, often for infractions that would not even warrant incarceration elsewhere.

Again, about 60% of those who pass through the penal courts receive alternatives-to-incarceration or time served. Of the 40% actually remanded to state prisons, the mean time served ‘ere parole or release is 30 months.

The people kvetching about ‘mass incarceration’ object to incarceration full stop.

Reply

23 Dzhaughn September 26, 2017 at 12:49 pm

To justify your recommendation, one needs an of the estimate the cost of incarceration.

The prisoner-year dollar cost is easy enough to estimate. Please also estimate the lost economic value of the prisoners activity, the cost to the family and friends of the prisoner, and the loss to all the members of society of living in a system so far from their notion of fairness and justice.

Reply

24 Anon7 September 26, 2017 at 8:29 pm

The mentally ill belong in psychiatric wards, but the ACLU doesn’t like that so they end up in a revolving door between prison and the streets, which is far worse. As for the remainder, perhaps they end up that way mostly because they have a habit of making bad choices. Perhaps you could host some of them and teach them what their single mothers and government schools did not.

Reply

25 The Other Jim September 26, 2017 at 10:24 am

SMOS gets it; Toby never will.

Reply

26 josh September 26, 2017 at 9:00 am

What do people think about corporal punishment or public humiliation? The latter especially, would probably be an effective deterrent, and isn’t really any crueler than locking a person up with a bunch of rapists and gangsters for years at a time.

Reply

27 MOFO September 26, 2017 at 9:13 am

We could try and make prison less rapey. Just a thought.

Reply

28 josh September 26, 2017 at 11:06 am

I assume if it was less rapey and assaulty in general, there would be less need to join a prison gang.

Reply

29 MOFO September 26, 2017 at 11:35 am

I assume so as well….? Not sure what your point is, though.

Reply

30 Steve September 26, 2017 at 12:07 pm

I think almost anything reducing people locked up in American prisons is going to be positive. Our prison system is so much more hellish than what is useful for prevention or deterrence — it’s a utilitarian nightmare.

Reply

31 Sam Haysom September 26, 2017 at 12:22 pm

I support the immediate release from prison of anyone that comes to live with you.

Reply

32 So Much For Subtlety September 26, 2017 at 8:20 pm

I am fine with that. Let’s hang all the murderers and anyone convicted of three or more felonies.

That would reduce the numbers locked up by quite a lot.

Reply

33 comments-af September 26, 2017 at 9:51 am

So incentives don’t matter?

Reply

34 Axa September 26, 2017 at 10:02 am

It depends.

As a 18 year old that kind of incentives were laughable. Today, near 40….incentives matter.

Reply

35 The Other Jim September 26, 2017 at 10:22 am

If people are going to commit crimes when let out of prison, you should not be letting them out of prison.

People who are going to commit crimes… should be kept in prison.

I realize that’s a radical take, so I’m just going to throw it out there, and try to let it sink in with you. In the meantime, we’ll just stick with your plan, which is never to put them in prison in the first place. Don’t worry — you will never have to live in a neighborhood where your plan affects you in the slightest way.

Reply

36 MOFO September 26, 2017 at 10:42 am

The only thing i would add to that is that we should endeavor to make prison a place that tries to make you into someone who will not commit crimes when you are let out.

I think the formula should be fairly straightforward: we will work hard to give everyone better options and opportunities than crime. If you choose crime despite our best efforts, then you can rot in a cell or wind up shot to death by the police. I dont think we are doing a very good job at the former.

Reply

37 pyroseed13 September 26, 2017 at 11:35 am

Yeah I agree with this. The problem is that incarceration is not cheap, and if some can be rehabilitated, then we should seek to do this. I don’t think there is anything wrong with offering educational programs for instance so that these people can become contributors to society. But it does need to be acknowledged that many criminals in prison are there for violent acts. Some of these are related to drug war, but some are probably just deranged killers. How many resources we should pump into those who commit very serious violent crimes is probably more of a moral question than an empirical one.

Reply

38 MOFO September 26, 2017 at 11:40 am

Id say its both and moral question and an empirical one. I should add that i would include people who are not yet in prison but at risk to commit crime. Offer them a better way before they go down the wrong path.

As for the deranged killers, you can shoot them for all i care. Prisons can be rehabilitative where appropriate, but they should also sequester and punish those unfit to remain in society as well.

Reply

39 Pshrnk September 26, 2017 at 11:28 am

@ The Other Jim “People who are going to commit crimes… should be kept in prison”

Well heck dude. Get em in the prison before they commit the first crime and we’ll be paradise.

Reply

40 MOFO September 26, 2017 at 11:36 am

Lame response.

Reply

41 Borjigid September 26, 2017 at 11:44 am

No, his response cut to the heart of the issue- can we know in advance who will commit crimes?

Reply

42 MOFO September 26, 2017 at 1:41 pm

“can we know in advance who will commit crimes?”

Not really, but we can still take action against those who do. What actions we take are more relevant than some vague hope to preemptively imprison those we think might cause crime.

43 Anonymous September 26, 2017 at 3:18 pm

Let me repeat the comment that is being responded to:

“People WHO ARE GOING TO commit crimes… should be kept in prison”

44 Borjigid September 26, 2017 at 11:47 pm

@Anonymous

Thank you

45 JonFraz September 26, 2017 at 1:39 pm

Get back to us when you find the precognitive kids from “The Minority Report”

Reply

46 mpowell September 26, 2017 at 2:03 pm

Why do so many responses assume the study is simply wrong? ‘Mandatory minimums are at margin increasing crime rates rather than reducing them? That can’t possibly be true! And if it is… well, just keep them locked up forever!’ So your solution is lifetime without parole for anything that would normally send a person to prison for any stretch of time? I mean, yeah, that would probably work at lowering crime. Even ignoring the questions of justice, is that really a remotely achievable policy objective? If not, then you really have to address the argument on its own terms – if longer sentences increase the crime rate on the margin, that’s a triple negative – higher crime, higher economic cost and more suffering for the convict. Even if you just care about one of the first two, the policy implications are clear.

Reply

47 ABD September 26, 2017 at 11:32 am

I’d love to see a program experiment with making replication a mandatory part of a PHD program. It’s probably too much to ask that the current standard of 3 papers be changed to 2 papers and a replication. But even if PHD students were forced to perform a replication as part of their advancement to candidacy, I think it could provide a number of benefits:

1. Provide PHD students with some very real practice in the nitty-gritty of research (tracking down data, dealing with dirty data, following methods). PHD students would also gain from possibly networking with the researchers they’re replicating.
2. Make replications more of a regular occurrence and hopefully reduce the defensiveness of some academics to the increased scrutiny. Related, get researchers in the habit of sharing data.
3. Improve peer review with the knowledge that someone might be “Monitoring the Monitor”
4. Help the field learn more about why/how replications fail (are researchers sloppy with data? Is it an over-reliance on P-hacking?)

I think that there might be other spill-over effects and likely some unintended negative consequences (longer time to graduate, issues with standardizing replications). And I’m not sure if matters if we define replication very narrowly (internal validity, same data => same results) or more broadly (external validity of key findings).

Reply

48 Anon7 September 26, 2017 at 8:46 pm

Perhaps it’s the old guys who are no longer doing as much original research who should also be given incentives to do it?

Reply

49 Joël September 26, 2017 at 11:57 am

That’s an idea. I think it could also be beneficial in Mathematics. Of course in pure math there is no data, and no “replication” properly speaking. But there are many ill-written papers around, where proofs are incomplete or sometimes wrong, which experts more or less know how to fix but that no one has taken the care to actually fix. Choosing one of those papers which is important and asking a grad student to rewrite it from scratch in a neat way would be the analog to ask an economics student to do a replication, and would be useful I think for the student and the community.

Reply

50 David Levine September 26, 2017 at 12:29 pm

I appreciate your reporting on the replication project as a success, not defending one of your own results or arguing with the overall interpretation. Bravo!

Reply

51 Li Zhi September 26, 2017 at 12:33 pm

Seems to me that there is “economic” value, “social” value, and individual Rights issues. What is the economic loss/gain for imprisonment? Surely the U.S. has a large enough population of ex-cons to determine whether they are a net economic contributor or drain. Social value is not adequately defined, but why not compare accomplishments between ex-cons and the rest of the population? My guess is that ex-cons contribute less, on average than say high-school drop-outs. (Of course, we can argue about their lack of post prison opportunities confounding any conclusions here). Aside from respecting their human rights, could it be that these people are, on average, a net cost for the rest of us? IDK. But with limited resources (which defines the subject(s) of economics, I believe) perhaps spending less on convicts and using that money elsewhere (schools, preschools, tech schools, healthcare, …) we’d all (except the prisoner who doesn’t get conjugal visits, cable TV, and free adult education (mostly about how to be a better criminal) along with free room and board) be better off.

Reply

52 Ray Lopez September 26, 2017 at 1:40 pm

David Roodman is a good researcher who reads Marginal Revolution he told me by email (I correspond with many famous Great & Good people, being prominent myself). Roodman wrote a critical report on microfinance called “Due Diligence: An Impertinent Inquiry into Microfinance” that concluded microfinance was perhaps more bad than good but was later made obsolete by another metastudy. But Roodman is a good scholar who does not seem to have an axe to grind.

Reply

53 Barbara September 26, 2017 at 3:16 pm

Alex – please tell us how you disagree with Roodman on these issues.

Reply

54 mulp September 27, 2017 at 5:18 am

“I don’t agree with all of Roodman’s conclusions but rather than pushing back I think it more important to underline how impressive the replication project is. ”

If you were a “climate scientist”, their conclusions you don’t agree with would be turned into at least one grant application each to gather data to challenge the conclusions you consider wrong.

That’s how the current list of papers cited in the big climate reports have grown to the thousands.

And deference and crime is a topic as huge as climate science. Theories in that field go back to the beginning of recorded history. After five to ten thousand years, many refuse to rule out the devil, demons, as the best theory of cause, with related cures and treatments.

Reply

Leave a Comment

Previous post:

Next post: