An update on the Reinhart and Rogoff critique and some observations

My previous post presented this:

Rortybomb summarizes it here, Matt Yglesias here, and the original paper is here (pdf), by Thomas Herndon, Michael Ash, and Robert Pollin.  I will read the paper soon.

I’ve now had some time to look at the paper, and here are a few observations:

1. I am of course open to publishing a rebuttal from R&R, but on a first read the authors make a strong case for their claim that the core Reinhart-Rogoff result — concerning the growth slowdown at debt at 90% of gdp — is based on a coding error and some data exclusion issues.  Please reread my earlier post on “the smell test.”

2. That said, as Ray Lopez mentions, including in the data the postwar bouncebacks of some Anglo countries (NZ, Australia, and Canada), as recommended by the critics, is not obviously going to improve the quality of the answer.  For instance the Kiwis have postwar growth rates of 7.7, 11.9, -9.9, and 10.8 percent, across the late 1940s.  Are those numbers — which were combined with high postwar levels of debt — relevant to current fiscal policy issues?  I say no, while admitting this may lead us to throw out other data points as well.  I don’t know what is the non-cherry-pick answer here or if there even is one.

3. It is perhaps unfortunate in this age of the internet that rebuttals must be presented so quickly, but so be it.  It will be interesting to hear from R&R.

4. Not too long ago I reread R&R to ascertain whether they actually present the 90% level as an emergency cliff of sorts.  I concluded they did not, although there were some sentences that a reader could take out of context toward confirming such an interpretation.

5. In the paper by the critics, the pp.7-9 discussion of “weighting by country” vs. “weighting by country-year” is very interesting, but the fact that it matters as much as it does makes me more skeptical about the entire enterprise.  Whether you should weight by population is important too.

6. I am seeing a large number of tweets which both misrepresent R&R or misrepresent their influence on current policies of “austerity.”

7. My own view, as you can read in The Great Stagnation, is that the primary mechanism is slow growth causing high debt/gdp ratios, not vice versa.  In any case this is by far the most important issue, whether or not you agree with my take on it.

8. The “case for austerity” didn’t rest much on R&R in the first place, rather on the notion that the bills have to be paid, dawdling on adjustment is not always so easy, and the feasible sum of international redistribution is quite low.  For this reason the UK should be relatively uninterested in immediate austerity and many nations in the eurozone periphery more interested.

9. In the blogosphere, the ratio of blog posts “attacking austerity” to “proposing constructive alternatives to austerity” is at least ten to one.  That too tells you something.  Many of the alternatives proposed would indeed pass a Benthamite cost-benefit test, at least if implemented as desired, but they are simply inconsistent with incentives and the relatively selfish nature of individual behavior.

10. The most interesting question to me is a rather squirrelly and subjective one: how should this episode change the relative ratios of what I read?  Should I in fact read fewer quantitative economics papers, instead (at the margin, of course) preferring more narrative history?  This is not the first time that an extremely influential major empirical result has been overturned or at least thrown into serious doubt.

Addendum: FT Alphaville weighs in.  And Annie Lowrey is tweeting some responses from R&R.


Is it typical for an econ paper like this to be published in a journal while keeping the data set secret?

Why do people keep making this claim about the data not being publicly available? It is:

and has been since at least November 2011:

What was kept "secret" was their computations. But the data was available for replication, Herndon et al. tried to replicate and couldn't, and it's good that information has come to light. So let's put this claim to bed.

Is data available in some format that doesn't have to be extensively munged first? Please don't tell me they did the analysis in Excel.

My colleagues Herndon Ash and Pollin provide estimates (using correct data) for a variety of time periods in their Table 5. This includes:
There is no evidence of a dramatic difference of the sort claimed by Reinhart and Rogoff for the "above 90 club" under any of the periodization.

Here is the data for from Herndon et al. for 1950-2009: below 30% is 4.1%, 30-60 is 3.0%, 60-90 is 3.1%, and 90+ is 2.1%. If you don't think a 1% difference in growth rates (between the 60-90 and 90+ categories) over a 60 year period is a "dramatic difference," you don't understand the power of compound growth. I recommend a textbook by... Cowen and Tabarrok.

Or here's a relevant example: if the US had grown 1% slower from 1950-2009, per capita GDP would be about half of what it is today.

"My own view, as you can read in The Great Stagnation, is that the primary mechanism is slow growth causing high debt/gdp ratios, not vice versa."

Oh ah. We should have chosen higher growth then, duh.

Seriously, perhaps it was Ponzi-like spending predicated on growth that failed to materialize. Bad idea.

And enough with this 90% cliff tipping point nonsense. Debt is bad, more debt is worse, too much debt is debilitating. The $17 trillion in US federal debt represents $17 trillion in slack we might have but don't, tipping point or not.

Debt is bad? Really? Does anyone actually, truly, deep in their heart, believe that debt is inherently bad (other than fundamentalist Muslims of course)? Corporations routinely increase returns by taking out debt. Students take on debt to pay for their education, thereby increasing their lifetime income and potentially go on to invent life-saving or life-improving technologies. Americans everywhere buy houses and cars by taking out mortgages and car loans.

Debt can be good, particularly so when interest rates are below the opportunity cost of capital.

"Debt can be good, particularly so when interest rates are below the opportunity cost of capital."

This has to be the most infuriating public finance argument out there. This is true when you're actually expecting to pay back debt. The relevant question is this: will debt taken today be paid back when it comes due or will it be rolled over into new debt. If the answer is the latter, then today's interest rates don't mean anything. It's future interest rates that matter.

Also, what if the interest rate is artificial and not what it should be in a true market? Is this not one of the main causes of the housing crisis? Sure a low interest rate looks great at the time, but if it is artificially low and is creating a bubble, then when that bubble pops the debt you took on is going to be really dangerous.

Agreed. Artificially low rates draw in the less educated and more easily duped people. Once the rates go up, their debt sky rockets unless they have a fixed mortgage. Most of the housing bubble was caused by artificially low rates in combination with IO loans and fraud by the mortgage agent since after the initial sale they were not held responsible for the stability of the mortgage.

IMO debt is only good if the long-turn payout is 1) better than if no debt was taken on 2) will be able to be paid within reasonable time. Debt is also good if you know how to manage it.

However, my honest opinion, long-term debt is bad. Really. I believe actually, truly, and deeply within my heart that long-term debt is bad and I am not a fundamentalist muslim. Long-term debt, unless the money borrowed is going towards infrastructure, is never good and should never occur. I think you need to go and look at some corporations books' a little more carefully, they are taking out short-term debt which is incredibly different than short-term. Furthermore, this would be fraud if these companies were taking out loans to boost their revenue for a quarter and then paying it back once the earnings are reported. To elaborate even further, this debt would still be looked at as a liability on the balance sheet and would not be seen as revenue.

You hit the nail on the head--the fact that the results can be erased with 49 vs 44 data points means the original result wasn't a result at all. But the question is, how did our statistical models lead us to *believe* it was a result? If we insist on continued use of statistical models with tremendous baked-in assumptions, we need much stricter p-value cutoffs when updating our beliefs. Alternatively--my preference--we need much much more widespread use of home-made, assumption-free permutation tests. What are the chances p<.05 with the original Reinhardt data set, even with the 44 points, if they just shuffled the countries between high and low debt? I bet slim.

Type I errors happen, and they're more common than nominally stated because of publication bias. This is well understood....

Nonparametric methods (such as the permutation test you propose) don't address this issue whatever.

But increasing the accepted *significant* p-value to 0.01 does.

It's a simple fact that p-values for tests that make assumptions about the structure of data will tend to be lower than those without. In fact that's the whole point of using them. This too "is well understood."

"Nonparametric" is a catchall for many methods that have underlying statistical models with built-in assumptions that just as with their parametric counterparts may or may not be satisfied and, often, it can never be *known* whether they're satisfied when working with smallish data sets. A permutation test makes no assumptions about the data, and when you're dealing with miniscule sets like these where the process generating the data is poorly understood, I really wouldn't trust anything else, at least not at the ridiculous "0.05" threshold.

And that's fine. The tradeoff is the power of your test. You'll make more Type II errors with a permutation test than you will with an appropriate parametric test. If you want to criticize the assumptions of the parametric test (which your second post nodded at), that's a legitimate critique. But your first point was about Type I errors, which (nominally) both tests have with the same frequency. It's no good to say that if you use a parametric model, you need a smaller p-value. What you need to do is justify your assumptions, and this should be done regardless of your alpha-level. I suppose you could use bad assumptions and require a lower p-value and get something approximating to an a higher alpha test with appropriate assumptions (at least in terms of Type I error). But then you're promising something your test isn't delivering by saying you're using one value for alpha but knowing that due to your bad assumptions, this is too low. Much better to just do sound statistical analysis with clearly stated and justified model assumptions and simple model checking methods.

this leads me to introduce
'Negative Marginal Product' workers aka economists.

please send me $5 to read more about this exciting new theory

Quoting Konczal:

"The U.K. has 19 years (1946-1964) above 90 percent debt-to-GDP with an average 2.4 percent growth rate. New Zealand has one year in their sample above 90 percent debt-to-GDP with a growth rate of -7.6. These two numbers, 2.4 and -7.6 percent, are given equal weight in the final calculation, as they average the countries equally... Reinhart-Rogoff don't discuss this methodology, either the fact that they are weighing this way or the justification for it, in their paper."

Indefensible, would expect to be flunked if I handed that in as a student.

If the party of science based their failure on one paper (they didn't of course, they just need a whipping post), then that's on them.

What's an acceptable alternative to austerity? Doing nothing? Delaying a solution until growth has returned? A combination of those two things--i.e. a consolidation of some sort, phased in some years down the line?

Also, why is austerity only acceptable in the form of cuts? Estonia, everyone's favorite example, increased its VAT.

I would like to add that many of these European countries have not actually cut spending, they are simply cutting the rate of increase. Reducing actual spending would do wonders as debt would be paid down and resources will be allocated more efficiently.

Debt would be paid down, assuming income remains constant. Any possibility that reducing spending would induce lower growth or even a recession, thus reducing income?

Or lets go to a doomsday scenario. What happens if austerity leads to price deflation? The real value of the debt goes up and then....

This is a very good post and yet it seems to me that you can do more than just adjust your reading habits.

The shift in economic research toward empirical studies and the quick cycle between working paper and policy impact are probably on net positive and regardless here to stay. That said economists and especially those at the interface with the policy/public sphere can do more to insure that the research is sound (not perfect, just sound). Three years later in the revise-and-resubmit at some journal is too late for serious scrutiny. I read a lot of empirical papers for my work and one of my go-to strategies is to read the guts of paper first and then 'write' my own intro/conclusion. There are some authors whose work I adore but whose overreach in their inference drives me bonkers. But they are responding to incentives...incentives that higher ups in the discipline and outsiders could help improve. A research paper should not be judged by its sound bites, incremental progress should be seen as progress too.

Next there has to be a way to introduce more openness of the data and methods. It is impossible not to make some mistakes and goofy assumptions when doing empirical work. I was taught to know what breaks all my models. And don't create "dinner parties" with my sample restrictions. And don't bury the bodies. That's hard to do and still get good publications or featured in popular press/blogs. Finally, proprietary data sets which help get you a top five publication are a big danger. The primary authors should negotiate access for a secondary author who gets published as an accompanying comment. Or come up with some other work around like invited a co-author with different views. But the main issue is that individual authors alone cannot change the equilibrium we are in, nor should they be punished solely for it.

PS One last substantive point...I totally agree that debt is an outcome first and a causal factor second (or maybe third). Too many discussion that focus on debt need some serious unpacking.

" And don’t create “dinner parties” with my sample restrictions. And don’t bury the bodies."

What does this mean?

for one of my first papers in grad school, I first made the sample so restricted: age, marital status, working, etc. that my professor said my sample looked like a dinner party ... I cut out too much of the interesting variation. Supposedly R&R were making important sample restrictions too. Bury the bodies means don't hide the odd assumptions, the intricate data open about what you did and why it is not perfect. empirical papers always involve compromise and inference...there is no perfect data set. those who pretend otherwise are not being honest.

Oh never mind. It appears from the next post that first-rate economists do not make mistakes...they are only misunderstood. No need to examine our research standards. Happy to be one of those third-rate economists who is clearly wrong.

Isn't this result an indicator of the upside of quantitative studies? Given access to the data, researchers could precisely and clearly critique the arguments without getting lost in semantics. I am interested in why Reinhart and Rogoff withheld their figures and methodology, and why other academics referenced their work without this information. Is this common procedure in academia, or were R & R granted the benefit of the doubt due to their academic stature?

Obviously the latter.

RR respond,

As expected. The question is, so if not 90, then what? R&R's response hints at why their critics didn't say it.

It's like when I heard about an hour of news reports about the marathon bombing without the only thing that mattered, the death toll.

Btw, this is what science is like folks. Get used to it. If this seems odd to you, then you might not be a scientist. Maybe you are just a science booster. It's kind of comical when a scientist wouldn't base a single prototype on a single paper (except in movies), let alone a macro-economy.

But usually in science people make their data public so other scientists can review their conclusions. And when someone notices an error (it happens), a scientist should admit it and correct it.

10. Don't read fewer quant. papers; just give them less weight when updating your beliefs. Since replication is the basis of empirical work, you might even consider reading more quant. papers, especially if it's difficult to not give too much weight to one well-known paper.

The response is out at WSJ:

Tyler, I understand that you are defensive. But it's moody affiliation.

The truth is that this is not science. The fact that economists even considered this paper is detrimental to the whole profession.

The result was not replicable. They didn't explicitate the weighting method. There were serious inverse causation problems that were not addressed. The groupings were arbitrary. No test of the effects of different groupings on the results. This would be enough for not taking them seriously.

Now we find out they deliberately excluded data points without explanation, the weighting method is crucial to their results, and there is a serious excel error.

This paper is a joke. It should never ever been discussed, even before this errors were discovered.

We should be ashamed of ourselves.

In short, it's too much like "climate science"?

No, it's too much like climate science denialism. A joke of second rate statistics errors and willful deceit passed off on an audience hungry for a friendly conclusion.

R&R's misleading paper in econ science is like Michael Mann's misleading "hockey stick" paper is to climate science. It's getting warmer...let the flames begin!

Ray, just out of curiosity, which statement, in your view, best characterizes what climate scientists think about the hockey stick? a) That it is wrong. b) We don't know if it is right or wrong. Or c) It has been replicated many times by many independent groups and is essentially right?

Or d) they selected the "appropriate" time span and - lo and behold - a hockey stick materialised. Much like R+R, methinks.

I have a similar take. It's not like the data set is huge or complicated. It boils down to just few hundreds of numbers. Given the simplicity of the data set, they should have understood this data inside out, especially as they were making a very bold claim. To learn that their result is not robust, that they excluded data without noting that they did so and without obvious good reason, and that they made a stupid mistake in their extremely simple code does not exactly inspire confidence in the profession.

re: "My own view, as you can read in The Great Stagnation, is that the primary mechanism is slow growth causing high debt/gdp ratios, not vice versa. "

Slow growth doesn't force government to spend too much, that is a choice. There is no reason for it to even maintain a constant % of GDP, in theory it could get more efficient if you forced it to like every other sector of the economy (aside from also limiting expansion of the tasks it performs and cutting them back). The share of GDP going to agriculture has decreased over the years, there is no apriori reason governments can't as well.. except they use baseline budgeting and dupe the public into thinking slower growth is a "cut". As this points out:
" The Federal government spent 3.7 times as much per person in 2011 as it did 50 years ago in 1961 when Democratic hero JFK was in office (adjusted for inflation). If Kennedy were around to propose his level of spending today he'd be denounced by the mainstream media as a radical extremist. State and local governments combined spent 4 times as much per capita. "

If you look at the data, a clear pattern emerges. Most of the time in the US, and around the world, the faster government spending grows, the slower the private economy grows (and vice versa) as this page shows:
It doesn't attempt to delve into what is cause and what is effect, though it offers some suggestions. The pattern is easily visible and easy to replicate to confirm without the need for lots of statistical analysis

Also re: "stagnation". Yes, the % yearly growth is slowing (and there needs to be a focus on ways to increase it since it depends on innovation and private investment in it).. but that is to be expected if it isn't exponential to begin with (as data shows it likely isn't, even if it matches exponential trends well, it matches quadratic trends even better), as this page points out:

" If Kennedy were around to propose his level of spending today he’d be denounced by the mainstream media as a radical extremist."

The same could be said about marginal tax rates.

But you're picking the last administration prior to the "war on poverty." Even our currently high rate of ~15% is much closer to the historical low of 11% than it is to the Kennedy era rates of around 22%.

The change in poverty rates from that era to now is most drastic among those 65 years and older, where it's dropped from nearly 30% to around 8 or 9%.

But I suspect you're right. I suspect that if a president proposed to drastically cut back gov't spending and tripling the number of seniors living in poverty, it wouldn't be judged kindly.


you might want to research Baumol's Cost Disease:

" The “case for austerity” didn’t rest much on R&R in the first place, rather on the notion that the bills have to be paid, dawdling on adjustment is not always so easy, and the feasible sum of international redistribution is quite low. For this reason the UK should be relatively uninterested in immediate austerity and many nations in the eurozone periphery more interested."

[Fine. It was not the first point of reference for the austerity proponents, but let's not minimize how several used this paper to support pro-austerity measures/arguments.]

How influential was the Rogoff-Reinhart study warning that high debt kills growth?
By Tim Fernholz

. . . Were other policymakers around the world paying attention? Absolutely:

“[I]t is widely acknowledged, based on serious research, that when public debt levels rise about 90% they tend to have a negative economic dynamism, which translates into low growth for many years.” — European Commissioner Olli Rehn.

“Economists who have studied sovereign debt tell us that letting total debt rise above 90 percent of GDP creates a drag on economic growth and intensifies the risk of a debt-fueled economic crisis.” — House Budget Committee Chairman and former Republican vice-presidential candidate Paul Ryan.

“It’s an excellent study, although in some ways what you’ve summarized understates the risks.”— Former US Treasury Secretary Tim Geithner.

“[W]e would soon get to a situation in which a debt-to-GDP ratio would be 100%. As economists such as Reinhart and Rogoff have argued, that is the level at which the overall stock of debt becomes dangerous for the long-term growth of an economy. They would argue that that is why Japan has had such a bad time for such a long period. If deficits really solved long-term economic growth, Japan would not have been stranded in the situation in which it has been for such a long time.” Lord Lamont of Lerwick, former UK chancellor and sometime adviser to current chancellor George Osborne.

“The debt hurts the economy already. The canonical work of Carmen Reinhart and Kenneth Rogoff and its successors carry a clear message: countries that have gross government debt in excess of 90% of Gross Domestic Product (GDP) are in the debt danger zone. Entering the zone means slower economic growth.”— Doug Holtz-Eakin, Chairman of the American Action Forum.

#5 seems exactly right. My take away from all of this is that there simply is not enough data to draw meaningful conclusions.

Which should make us wary of anyone trying to do so.

I am surprised that there is a serious discussion about this. How can high debt levels not slow down growth. For that to be so, the projects that the borrowed funds were invested in would have to provide returns higher than the cost of capital. What modern government spending program can boast that.

1. Your comment appears to assume that investment and capital accumulation are the source of growth -- the past few decades of empirical papers on the subject, however, cast serious doubt on the idea that investment and capital accumulation play such an important role.
2. To the extent investment plays a secondary role in growth, there is no evidence of 1:1 crowding out. $1 of government borrowing does not displace $1 of private investment and, in the aftermath of a very serious financial crisis and recession, it might be tough to find any evidence of crowding out at all. This is also not very controversial as an empirical matter.

For me, this results from two peculiarities about academic publishing in Economics: (1) The lack of blind reviews, and (2) the very common practice of using "benign" reviewers for articles the editor wants.

No other discipline is like this and if these had been in place, someone would probably have caught these errors or similar errors in Reinhart and Rogoff's 2002 work on de facto exchange regimes.

So what? Government debt does not slow growth but with population not growing, why use debt? You can only tax a population so much even if per capita real income is growing at a good pace.

The United States survived 17% annual inflation under Carter. Well, two years of 17% inflation turns a .9 debt/GDP ratio into a 65% ratio, as long as the debt is basically domestically held.

No doubt the Chinese might start looking for inflation-adjusted bonds -- but they'd probably take inflation-adjusted zeroes.

So basically this whole thing is Republican talking points, nothing else.

* * *

Now then, can we talk about something serious for a change: in a services based economy, how come the US doesn't have a Canadian-style Goods-and-Services tax?

Tetchy-tetchy, I know: Bobby Jindal just committed hara-kiri with a sales tax proposal, but that doesn't change the fact that it's utterly unavoidable at some point.

Sales point to the Right: it's a flat tax. (It isn't really, but it's possible Jindal's staff were stupid enough to think that it was.)

Sales point to the Left: consumption taxes are the only way you can tax the ninth decile. These are the "New Class" of Milovan Djilas's good and ever truer naming; they are the major consumers and the major producers of information and other services. This slice of taxpayers includes most staff level accountants and lawyers -- the major producers and consumers of legal, however devious, tax evasion. They pay what they feel like paying in income taxes, but they don't got no choice about a well constructed consumption tax.


I think a lot of people are forgetting that the markets were not nearly as interconnected nor informational as they are today. The markets impacts happen almost instantaneously now, where it would take some time in the past for certain actions to have a ripple effect. These debt ratios are also different because of the circumstances that helped accrue the debt. For instance, post WWII was infrastructure recovery spending with the countries involved. Today, the US is in its hole because of conflict spending (we are not at war, Congress never declared war), financial bailouts, and only a small portion is because of infrastructure spending. The difference is that the long-term debt for infrastructure would bring in long-term gains since businesses and consumers will have an easier way of doing business and delivering necessary supplies.

If this isn't the death blow to using excel in quantitative analyses, I don't know what it would take. This mistake simply wouldn't happen with python, R, SAS, or what ever scripting language you want to use. I'm stunned such esteemed researchers still use it.

Hiya very nice web site!! Spent an interesting few minutes looking about.
I'm satisfied to seek out numerous useful information right here within the post, we want develop extra techniques on this regard, thank you for sharing. . . . . .

Every weekend i used to pay a visit this website,
as i wish for enjoyment, as this this site conations really nice funny data too.

Thanks for one's marvelous posting! I definitely enjoyed reading it, you will be a great author. I will make certain to bookmark your blog and will eventually come back later in life. I hope you continue your great posts, have a nice day!

Hello! I simply want to give a huge thumbs up for the nice info you have right here
on this post. I will probably be coming back to your blog for extra soon.

Nice post. I used to be checking constantly this weblog and I'm impressed! Extremely helpful info specially the final phase :) I care for such info a lot. I used to be looking for this certain info for a long time. Thank you and good luck.

hey there and thanks for your info – I’ve definitely picked up anything new from right here.
I did then again experience some technical issues using this
website, as I skilled to reload the web site a lot
of times previous to I may get it to load properly. I have been brooding
about if your hosting is OK? Now not that I'm complaining, but slow loading cases instances will sometimes affect your placement in google and injury your quality rating if advertising and ***********

I'm not sure why but this blog is loading incredibly slow for me. Is anyone else having this issue or is it a issue on my end? I'll check back later on and see if the problem still exists.

Comments for this post are closed