Advice for Graduate Students

In a video at Freakonomics Steve Levitt talks about the genesis of the idea for his abortion and crime paper with John Donohue.  The key sentence, "and so I spent the first couple of years of research…".

In a unrelated post he also tells us this, "back in grad school, I had carpal tunnel problems from entering too much data…"

Learn from the master, grasshopper.


the key sentence is of course "completely fortuitously..."

Widely debunked? No. But the critique papers have found significant, if minor, errors that weaken the conclusion quite a bit. So I think it depends who you ask.

i've read a failed attempt to debunk the abortion-crime link but perhaps there are better debunkers out there.

either way, levitt has done a lot more than just abortion-crime.

check his CV.

That really isn't making me look forward to attending graduate school......

Here's an excerpt from Jim Manzi's review in National Review comparing Levitt's Freakonomics to Lott's Freedomnomics, and noting some fundamental similarities between them:

"The second weakness of Freedomnomics is more fundamental, and is shared with Freakonomics: Both authors oversell their ability to analyze society mathematically. They repeatedly fall into the trap of mistaking correlation for causality.

"Levitt's most notorious conclusion in Freakonomics was that a significant fraction of U.S. crime reduction in the 1990s was actually caused by changes that Roe v. Wade set in motion 20 years earlier. The primary evidence offered for this in the main section of his book was that the five states that liberalized abortion laws prior to Roe experienced a crime reduction prior to the states that did not. Presumably recognizing that this is not exactly the Michelson-Morley experiment, he then tried to buttress his conclusion by referencing ancillary, even less persuasive, analyses: one showing that the states with the highest abortion rates in the 1970s experienced the greatest crime drops in the 1990s, and a second showing that for states with high abortion rates, the entire decline in crime was among the post-Roe cohort. Finally, he magisterially stated that Australia and Canada have seen similar results.

"Lott devotes considerable space in Freedomnomics to attacking Levitt's analysis of this issue. Lott slightly changed the data set that Levitt used in his relevant academic papers, through such adjustments as more closely linking the date of a murderer's arrest with the date of his crime, and better accounting for abortions that occurred prior to legalization. When the same methodology was applied to these data, Lott found that more abortions appeared to increase crime. Lott also references an independent paper by two Federal Reserve economists that demonstrated Levitt had failed to conduct a specific analysis that he had asserted was done in his most prominent academic paper on abortion and crime. Once this methodological error was fixed, it appeared that abortion had very little effect on crime one way or the other. Levitt responded to this criticism by further slightly changing his own data set, primarily by estimating the degree to which people moved between states in order to track effects across state lines. Not shockingly, he found that his original relationship partially reappeared when he used this reworked data set.

"Levitt wrote that Roe is "like the proverbial butterfly that flaps its wings on one continent and eventually creates a hurricane on another." He ought to be more careful with his similes: Surely he knew that he was echoing meteorologist Edward Lorenz's famous evocation of a global climate system--one that had such a dense web of interconnected pathways of causation that it made long-term weather forecasting a fool's errand. The actual event that inspired this observation was that, one day in 1961, Lorenz entered .506 instead of .506127 for one parameter in a climate-forecasting model and discovered that it produced a wildly different long-term weather forecast. This is, of course, directly analogous to what we see in the abortion-crime debate: Tiny changes in the data set yield vastly different results. This is a telltale sign (as if another were needed) that human society is far too complicated to yield to the analytical tools that Lott and Levitt bring to bear. Nobody in this debate has any reliable, analytically derived idea of what impact abortion legalization has had on crime.

"This is not an isolated example; in fact, such analyses are essential to both books."

So the advice is "datamine until something pops out at you"

Hmm that sounds less pejorative than I was hoping for...

"When the legend becomes fact, print the legend"

The analysis of Donohue and Levitt, as well as parts of the book “Freakonomics†, has been shown to be faulty. The work was presented at the Conference of Empirical Legal Studies at NYU law school ( by Prof. Anderson of Cornell University. Here is an abstract of the paper:

Concepts of numerical analysis with applications to least squares problems are introduced in a manner which the practitioner can readily apply to their research problems, especially in the social sciences. Numerical analysis is mainly concerned with the accuracy and stability of numerical algorithms. We frame these concerns in terms of forward and backward error, two important concepts in helping to understand the quality of the computed answers. The goal of numerical computing is to get correct, approximate answers to the true solution. We extended this forward and backward error framework to issues in least squares problems and check the condition of the regression problem via condition numbers. The more ill-conditioned the data are, the more sensitive the computed solution is to perturbations in the data, and the more unstable the computed solutions become. Condition numbers can also be used to signal the presence of solution degrading collinearity in regression problems. We apply the various numerical analysis tools outlined with some model diagnostics to the abortion-crime debate, and show the regression analysis used in various papers addressing the abortion-crime debate cannot be trusted.

Comments for this post are closed