Reinhart and Rogoff respond

I knew something would come quickly, though this is quicker than I had expected.  Via Matt Yglesias:

We literally just received this draft comment, and will review it in due course. On a cursory look, it seems that that Herndon Ash and Pollen also find lower growth when debt is over 90% (they find 0-30 debt/GDP, 4.2% growth; 30-60, 3.1 %; 60-90, 3.2%,; 90-120, 2.4% and over 120, 1.6%). These results are, in fact, of a similar order of magnitude to the detailed country by country results we present in table 1 of the AER paper, and to the median results in Figure 2. And they are similar to estimates in much of the large and growing literature, including our own attached August 2012 Journal of Economic Perspectives paper (joint with Vincent Reinhart) . However, these strong similarities are not what these authors choose to emphasize.

2012 JEP paper largely anticipates and addresses any concerns about aggregation (the main bone of conention here), The JEP paper not only provides individual country averages (as we already featured in Table 1 of the 2010 AER paper) but it goes further and provide episode by episode averages. Not surprisingly, the results are broadly similar to our original 2010 AER table 1 averages and to the median results that also figure prominently. It is hard to see how one can interpret these tables and individual country results as showing that public debt overhang over 90% is clearly benign.

The JEP paper with Vincent Reinhart looks at all public debt overhang episodes for advanced countries in our database, dating back to 1800. The overall average result shows that public debt overhang episodes (over 90% GDP for five years or more) are associated with 1.2% lower growth as compared to growth when debt is under 90%. (We also include in our tables the small number of shorter episodes.) Note that because the historical public debt overhang episodes last an average of over 20 years, the cumulative effects of small growth differences are potentially quite large. It is utterly misleading to speak of a 1% growth differential that lasts 10-25 years as small.

By the way, we are very careful in all our papers to speak of “association” and not “causality” since of course our 2009 book THIS TIME IS DIFFERENT showed that debt explodes in the immediate aftermath of financial crises. This is why we restrict attention to longer debt overhang periods in the JEP paper., though as noted there are only a very limited number of short ones. Moreover, we have generally emphasized the 1% differential median result in all our discussions and subsequent writing, precisely to be understated and cautious , and also in recognition of the results in our core Table 1 (AER paper).

Lastly, our 2012 JEP paper cites papers from the BIS, IMF and OECD (among others) which virtually all find very similar conclusions to original findings, albeit with slight differences in threshold, and many nuances of alternative interpretation.. These later papers, by they way, use a variety of methodologies for dealing with non-linearity and also for trying to determine causation. Of course much further research is needed as the data we developed and is being used in these studies is new. Nevertheless, the weight of the evidence to date -including this latest comment — seems entirely consistent with our original interpretation of the data in our 2010 AER paper.

Carmen Reinhart and Kenneth Rogoff
April 16, 2013

Addendum: Paul Krugman comments.


A chess analogy is appropriate here (Rogoff is a grandmaster). When a GM blunders --and it is rare--they defend very tenaciously. I have played a master who hung a piece and still beat me easily (and I'm above average in strength). Same with Rogoff--don't expect them to just roll over.

Posting as Ruy Lopez might bolster the analogy.

Well, math is not a game. And if economics is a science, then recognizing errors is critical in achieving accuracy.

But at least we now have a lot of baseline data on what happens when policy is based on flawed data foundations.

What baseline data would that be?

The last five years, of course. The UK being a fine example, where the IMF has warned that austerity is going too far.

'George Osborne has been told by the International Monetary Fund to rethink the pace of his deficit reduction plan after the Washington based institution cut its forecast for UK growth in both 2013 and 2014.

The IMF's flagship publication – the half-yearly World Economic Outlook – provided fresh ammunition for the chancellor's Labour critics when it said the Treasury should contemplate being flexible about its austerity strategy.

In its latest forecasts, the Fund said gross domestic product in the UK would rise by 0.7% this year and by 1.5% in 2014 – in both cases a cut of 0.3 points from its last set of predictions in January.'

The article is fairly interesting - though not exactly surprising.

But that doesn't say anything about highly-indebted countries. The UK government isn't highly indebted.

So the UK only borrowed 7.9% of gdp instead of 9.5 (2012 - 2011).

Projections are they will not even try austerity in the near future.

"– though not exactly surprising." Nope. Not surprising, but not in the way you intended.

Scientists are notoriously bad at giving up their pet theories -- the field progresses as people die.

That being said, Reinhart and Rogoff's paper has been *the* basis for a policy decision that they fully endorsed -- which has led to the collapse of the world economy and increasing levels of both poverty and (inevitably) civil strife. And their paper has now been shown to be flawed -- and, based upon the types of analytical errors, probably was written in bad faith.

That's not a theoretical debate. That's flat-out charlatanry. That's Andrew Wakefield fabricating a study connecting autism and vaccines -- resulting in measles outbreaks that can kill children. Except that analogy isn't even serious enough -- this isn't an outbreak on the level of measles but of *smallpox.*

Wakefield is (rightly) shunned by the medical establishment -- and virtually all professionals who were tied to his (and related) publications have publicly disavowed their connection. The fact that Reinhart and Rogoff's colleagues are still willing to be *seen* with them says volumes about the ethical practices of economists.

Stalin was right. A million deaths *is* a statistic.

None of that will save Rogoff if he's a patzer of a data scientist.

Unless I missed something, R&R didn't respond to the criticisms about their methodology. Namely, why did they exclude major countries with over 90% debt/GDP ratios that had higher growth rates, and why did they weight by country rather than country-years? Although this seems to be only a preliminary response, failure to answer these questions would be a tacit admission that their methodology was somewhat arbitrary and weakens their conclusion.

Seems likely to follow exactly the same trend as complaints about methodology in climate change papers. We've got complaints about the original data and models not being open for easy replication, replies that other papers (plus the original paper with a corrected method) still show the same basic result, most of the people on the "anti" side conceding that "eventually" we'd have a problem, but there's no need to address it in the short run, etc.

To point out the obvious, this is different from climate change in that climate change on the scale we are concerned about is unprecedented in human history and could potentially end human life on the planet, whereas high levels of government debt have been seen repeatedly, and while growth of -0.1%, to take the low side, would not be pleasant, it would hardly mean the end of the world.

"could potentially end human life on the planet"

No. That is all.

And remember the last depression. That seemed to have some repercussions.

"unprecedented in human history " ... er, except all those other time when it was worse.

To point out the obvious, 'climate science' deals with a poorly-understood phenomenon, with the most outrageous prescriptions demanding drastic changes today that will have a miniscule impact a century from now on CO2 levels, while the pages of history are replete with stories of spendthrift governments collapsing due to debt addiction.

With this politically motivated attack on Reinhart and Rogoff the forces of darkness and irrationality are engaging in another round of Science denialism.

good one lol

The problem has never been with the science, such as it was. Or, rather, there are always problems with the science, as with all science. Scientists are people. They are not especially rewarded for being ethical people. Though, they, at least the ones from ethically oriented cultures, may start out as high ethics people before joining science. The real forces of darkness are in the economics and policy sphere.

and, as he said, others found "many nuances of alternative interpretation.."

Sounds like weasel words.

Nuancively speaking.

I don't know how nit-picking this is, but it seems to be that a better analysis would be to pick multiple numbers, not just 90, to be sure that 90 isn't cherry-picked. Where did this 90% number come from? Is it some famous equilibrium point (seems unlikely to me)? It seems like a multitude of factors should affect where an overhang occurs, including technological progress (as Solow would define it). Do a nation's assets matter?

Personally, I do suspect that for any given economy, there is a point at which debt is dangerous, but this doesn't strike me as the most complete reasoning on the matter.

Seems to be a common occurrence in economics: Somebody works their fingers to the bone to develop a NEW dataset, uses it to draw some conclusions, exhibits some weaknesses along the way -- perhaps a deficit of theory, perhaps poor exploitation of econometric techniques, and YIKES, a coding error -- and the authors are deemed stupid.

Anybody in the field NOT wishing to work with the dataset? Anybody in the field adding to the dataset? 'Course not.

Why bother with new systematized facts at all, when there are so many old facts around to re-analyze for the 1000th time? Tells you something about incentives in the academy.

"if you look at it, you see that most of the apparent relationship is coming from Italy and Japan" (Krugman)

Is Krugman being honest here? Perhaps he should share his data set. It looks to me like the U.S. and France also show the correlation pretty strongly. At the very least, this isn't easy to eye-ball.

"is Krugman being honest"

With Krugman, the truth is elastic.

"Is Krugman being honest here?"

There is always a first time for everything.

Probably why he was given a Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel.

His approach to the truth being in the finest of company in that esteemed list.

Ha! Maybe. It is somewhat amusing how people are debating whether R&R's paper was peer reviewed. What Warren Buffett says about CEOs is you need energy, intelligence and integrity. Most of us deal with people with hopefully energy and integrity. Which of those, in order, do you think academia rewards for? Which comes in last, I'll give you three guesses and the first two don't count.

I agree with you: I see the same thing as you about the U.S. and France (and even more so, Germany!) and the correlation situation there is, if anything, even clearer than Italy or Japan. But the big R&R controversy concerns the headline claim, about the importance of being over or under the 90% debt line. In terms of thinking about _this_ magic number, which maybe Krugman was (though to be as charitable as possible, it isn't super clear) doesn't his point still stand?
I don't know if there is a serious dispute that debt is on the whole statistically correlated with lower growth (if you consider all ranges of debt, and you stick with correlation rather than causation). But there is a suspicious that once you go above 90% debt, this will have low growth implications (a magic threshold story, plus a causal claim). Does that hold up at all in light of this data? In the tiniest?

I'll answer your last question(s) by pointing to my other comment a few comments back.

It strikes me as silly that this study is considered of such importance to begin with. There are better ways of going about the discussion as to when debt becomes excessive and slows growth. Personally, I think it's far easier to just say that taxes slow growth by taking capital away from market participants, then linking the need to raise taxes to high debt. In a vacuum, high debt seems like an overly simplistic variable subject to political warfare.

Taxes raise growth because they re-direct resources that would be spent on short term consumption to investments in basic research that produce long term growth. Too simplistic? Well, so is "taxes slow growth by taking capital away from market participants"

Please notice that for these other countries there are no episodes of high debt

While I agree with your interpretation of Krugman's figure (it does look like a number of the countries are showing the relationship), he did state exactly where he got the data. It's a publicly available data set, unlike the R&R's. It's easy enough to go get it and graph it yourself: Not as easy as just speculating and Krugman trolling, I suppose...

Oh, I was being cheeky about the data. But I do disagree with his interpretation of causality in Japan. The problem with Japan is that its modern financial history is too murky. That's what led the banks to merge in the late 90's: took much cooking of the books for too long. Japan is very much askew from better data sources.

If I have time, I may take a look at the data, but time is currently thin and this particular battle doesn't leave me terribly motivated since this looks like a silly fight where the only sure thing is that everyone is going to justify their position.

Can somebody please tell me where the debt data is on the Conference Board's website? I've looked at all the files I can see and that variable is not in any of them. Please email me the link if you know where the data is. The link above was in Krugman's post but it does not get me to any debt data.

To a layman (i.e., me), this just seems like everyone involved is defending their priors. Hopefully this is because I do not understand the situation.

You are right on, from another layman. Their side never believed R&R in the first place, and my side didn't need R&R to tell us what we already believed.

Economics is the queen of the social sciences. Economics is the queen of the social sciences. Economics is the queen of the social sciences. Economics is...

Yes it is: Only relative prices matter! :-)

ISWYDT. very nice.

Does the fact that the R&R errors are so simple make it more or less probable that they were committed intentionally? Does Hanlon's Razor - never attribute to malice that which is adequately explained by stupidity - apply here?

In other words, were the cherries picked accidentally or on purpose? And was the pie baked with those cherries simply a coincidence? Well, here is a bit of information from Dean Baker -

‘The most important of these errors was excluding four years of growth data from New Zealand in which it was above the 90% debt-to-GDP threshold. When these four years are added in, the average growth rate in New Zealand for its high debt years was 2.6%, compared to the -7.6% that R&R had entered in their calculation.

Since R&R country-weight their data (each country’s growth rate has the same weight), and there are only seven countries that cross into the high-debt region, correcting this one mistake alone adds 1.5 percentage points to the average growth rate for the high-debt countries. This eliminates most of the falloff in growth that R&R find from high debt levels. (HAP find several other important errors in the R&R paper, but the missing New Zealand years are the biggest part of the story.)’

Maybe it is a little more subtle than that.

Surely, such a (admittedly huge) mistake that can be discovered, understood and disseminated so quickly and widely is a "simple" mistake? If R&R wanted to intentionally screw up, would they not have chosen a clever, ingenious, perhaps mathematically complex way of doing so? This is not only a stupid mistake, it is also a stupid way - given the exalted reputations at stake - to intentionally make a mistake.

The problem now for R&R is that:
i) if they admit that there is a cock-up, they come across as bumbling idiots who can't do arithmetic
ii) if they don't, they come across as politicized idiots who don't let data get in the way of ideology

If they had insted made a fancy econometric mistake to force a result, then
i) they just admit that they unintentionally made a mistake in the complex math, and maybe people forgive them

so by backward reasoning, perhaps it is indeed an honest error?

I am not suggesting it could not be (a series of) mistakes. There is a much better discussion of this at

But I will quote one comment from there (not mine) - 'I suppose that it's impossible to absolutely prove that they did this deliberately. The fact that they have multiple significant errors in a single short analysis and one of them just involved "some data magically vanishing" (which is the number one classic way to bias a result) seems like astonishingly bad performance for professionals in the field. The fact that the errors systematically moved the results in one direction is also dodgy - generally, errors tend to cancel somewhat - but it's not proof.

So, yes - you can't rule out massive levels of incompetence and chance. Perhaps the error rate when economists make simple calculations is so very great that multiple errors will inevitably slip in, even with two authors reviewing each others' work. I don't believe it, but yes, there's plausible doubt...
posted by lupus_yonderboy'

Thanks, that's a very interesting thread.

The corrected figures obviously just reflect that slow growth causes debt to gradually pile up. R&R are not going to be able to talk their way out of this--their theory was premised on the idea that there is some magical debt threshold which makes economies go off the rails.

Since R&R, by all indications, are not charlatans, I think this incident really speaks to the sloppiness of mainstream, top-tier eCONometrics. This probably could only happen if cherry-picking cases and occassionally miscoding data were not considered life-or-death errors in the discipline.

I must have missed the bit where they thank Herndon et al for their review of the analysis and reporting the bugs and other problems it uncovered.

At the most basic level, the Excel blunder is a reminder that even the smartest people routinely make simple mistakes when they write software. So we should assume that there will be mistakes in the early versions of any software-based analysis and heavily discount results that are not accompanied by basic quality-assurance checks like available data, runnable code and tests, visible reviews, etc compared to those that are. This is part of message of the reproducible research program, covered in e.g. "the case for open computer programs" in Nature last year. It's remarkable that this has not become the norm in the scientific community by now given that arxiv started in 1991, although looks like an interesting attempt to support this.

R&R are not being entirely truthful when they claim that: "we are very careful in all our papers to speak of “association” and not “causality”".

In their JEP paper, they write that:

1. "The long duration [of the public debt overhang/low growth association] belies the view that the correlation is caused mainly by debt buildups during business cycle recessions." I.e. they clearly imply that the correlation at least partly comes stems from debt overhangs causing slower growth.

2. "These growth-reducing effects of high public debt are apparently not transmitted exclusively through high real interest rates". How is this not claiming causation from high public debt to slower growth?

Why is that not being careful? The whole point is to determine the causation. If that is their conclusion, they can state it carefully. The point about why they should be careful is that, first, that is part of the process of separating correlation from causation and secondly because debts rise ("soar") during financial crises, which they say they point out and say they make clear that this is not the part they are referring to.

R&R say that they make no claim about causality. But they do make claims about causality, for example when they talk about "These growth-reducing effects of high public debt" as if they have established them as facts. And if you read their JEP paper there are other such statements about causality, e.g. "at a debt/GDP ratio around 90 percent of GDP, public debt overhang does seem to have a negative effect on growth.”

I don't think so. They say they are careful when they are talking about causation versus correlation. The whole point of looking at correlations is to determine causation. So, if they discuss enough correlations that they conclude a causation, they can do that carefully, as should everyone.

No, R&R do not say that "they are careful when they are talking about causation versus correlation". They say that they "are very careful in all our papers to speak of “association” and not “causality”, i.e. they say that they only make claims about correlations and not about causation. But as I pointed out, they do make claims about causation.

And no,the point of looking at correlations is not to determine causation. You can look at correlations as much as you want but It is not going to help you determine causation.

And this:

"the weight of the evidence suggests that a public debt overhang does slow down the annual rate of economic
growth" (p.84 of their JEP paper).

This sure looks like a claim about causation from public debt overhang to slower growth.

Because that is the whole point, to determine if that is true. They are in fact still working on it, they say.

What's so hard about all this?

The more debt you have, the more interest you are paying (more than linearly, since you'll pay a higher rate since with a high debt you are less likely to repay).

Obviously, this means higher taxes than other nations, so people will be more likely to produce in other nations, reducing GDP.

Also obviously, there is no "magic number" "low growth" threshold, it gets continuously worse and worse until you default. No economics degree required.

Shhh. A branch of the profession is dedicated to the idea that a dollar of debt, via magical hand-waving, only costs ninety cents. This has its roots in Marx, of course:

"The nickel today is not what it was fifteen years ago. Do you know what this country needs today?…A seven-cent nickel. Yessiree, we’ve been using the five-cent nickel in this country since 1492. Now that’s pretty near a hundred years’ daylight saving. Now, why not give the seven-cent nickel a chance? If that works out, next year we could have an eight-cent nickel. Think what that would mean. You could go to a newsstand, buy a three-cent newspaper and get the same nickel back again. One nickel carefully used would last a family a lifetime!"

That part is obvious. But Keynesianism makes the margins critical.

One of the nice things about economics, is if you don't like somebody, feel free to not believe their conclusions. On that score, it may be even better than science.

If you don't like OUR paper, then it's your responsibilty to go create YOUR paper. Har har.

Comments for this post are closed