DSGE has Failed the Market Test

by on January 10, 2014 at 9:34 am in Economics | Permalink

Noahpinion makes a good point:

As far as I’m aware, private-sector firms don’t hire anyone to make DSGE models, implement DSGE models, or even scan the DSGE literature. There are a lot of firms that make macro bets in the finance industry – investment banks, macro hedge funds, bond funds. To my knowledge, none of these firms spends one thin dime on DSGE. I’ve called and emailed everyone I could think of who knows what financial-industry macroeconomists do, and they’re all unanimous – they’ve never heard of anyone in finance using a DSGE model.

If you know someone who does, please reply in the comments. I’m sure there’s someone out there. But even if there is, they haven’t soared to fame and fortune on the back of their DSGE model.

…In other words, DSGE models (not just “Freshwater” models, I mean the whole kit & caboodle) have failed a key test of usefulness. Their main selling point – satisfying the Lucas critique – should make them very valuable to industry. But industry shuns them.

1 Willitts January 10, 2014 at 9:39 am

Ever consider that nearly everything in macroeconomics is bunk?

What is there that is solid?

2 Rich Berger January 10, 2014 at 9:45 am

Good question – what are the solid, demonstrable successes of macroeconomics?

3 JWatts January 10, 2014 at 10:00 am

Thick, text books. Not as successful as Law or Med, but still they sell a lot of books.

4 Ricardo January 10, 2014 at 10:33 am

Economics is definitely the worst of the social sciences, except for all the others.

5 Willitts January 10, 2014 at 10:48 am

I would argue that deposit insurance is a good policy derived from a macro observation.

Elastic money supply smoothed out money and credit problems in agriculture.

Tightening money supply to combat inflation seems to work, thus confirming at least some part of the QTM.

I think we know better what will wreck an economy than how to fix it, and fine tuning is a sham.

I’m skeptical of any theory that tells politicians what they want to hear.

6 Govco January 10, 2014 at 1:07 pm

I think we know better what will wreck an economy than how to fix it, and fine tuning is a sham.

This is regularly said of baseball managers– their game strategies reduce chances for success more than help. But as the old, orthodox GMs, managers and, critically, reporters disappear from the game, the new crop improves game strategies (e.g., less sacrifice bunting). By analogy, then, once orthodox macro professors*, their political consumers, and, critically, reporters disappear, macro can progress and improve (e.g., less DSGE).

*A note that may only interest me: Galileo’s main antagonists were University professors, preserving academic orthodox astronomy against this foppish, popular, biting layman’s criticisms. The Church intervened when those professors retreated from astronomical debate into bad-faith allegations of heresy and initially cleared Galileo. Once more, High School History must be undone.

7 gOVCO January 10, 2014 at 2:15 pm

I would argue that deposit insurance is a good policy derived from a macro observation.

Maybe not, deposit insurance eliminated any competitive reason to maintain a fat capital cushion AND incentivized banks to leverage up on junk assets. Like this: Deposit insurance required the FDIC to regulate insured depositories (mind its liabilities). The FDIC imposed capital requirements, defined by reference to Basel I and modified by the “Recourse Rule”, that differentiated risks: 10% for commercial loans, 5% for mortgages and an enticing 2% capital cushion for ASBs (i) rated AA or AA by government approved ratings agencies or (ii) issued by GSEs like Fannie and Freddie. Oops!

8 guest January 10, 2014 at 4:27 pm

Its unclear whether FDIC did that. On one hand, we have the historical fact that Banks fail less often these days. On the other hand we have a historical fact that in Canada banks also fail less often and they are a TBTF oligopoly.

9 Anon. January 10, 2014 at 12:50 pm

Ray Dalio, surely? He has done very well with his quant macro approach.

10 mulp January 10, 2014 at 1:26 pm

Job killing tax cuts have been made a proven method of trying to create jobs given the repeated failures to create jobs with tax cuts that are followed with failed job creating tax cuts.

Now the argument is that government spending at a time of record levels of excess labor supply is depriving the economy of labor which is thus hindering growth – if government funded jobs are ended, then the lazy incompetent workers who can’t make it in the private sector will be liberated to become the flood of Bill Gates and Steve Jobs needed to create jobs by inventing labor saving high tech, or by going out and mining labor saving oil and coal, because the worst thing for an economy is building productive capital because building capital requires labor, and all labor is a drag on the economy if you must pay them a living wage – onlyl when labor qualifies for welfare are wages beginning to be low enough, but welfare creates dependency so workers need to be forced to find a way to work without spending any money on living.

After all, if labor demands they be paid enough to live, then they will be replaced by robots who simply live at work.

When stock prices go up, shareholders can borrow because they are wealthier, and then consume the production of the robots. In the idea economy, no dividends or interest are paid to everything results in capital gains which makes shareholders richer and thus fund consumption by increasing debt because increasing debt increases capital gains, and if everyone is an owner, then no workers are required because robots can do all the work, and profits will be higher and capital gains higher meaning debt can be higher to fund increased consumption. All without the economic drag of labor and wages.

11 mpowell January 10, 2014 at 9:55 am

Its apparent to anyone not professionally invested in DSGE that they are useless. Their results are highly sensitive to unknown (and probably unknowable) details in their microfoundations. How could you ever use something like that to do anything but show macroeconomics is hard? Academic economists fell in love with them because they allowed them to feel like were explaining things in a rigorous way. Turns out, not so much. There also seems to be a weird intersection of political preferences and academic study that is hindering the academies ability to absorb the clear lessons of decades of DSGE failure and attempt to move on to more useful avenues.

12 PJ January 10, 2014 at 10:08 am

Businesses care about making forecasts of very specific variables and I think there are much simpler and equally or more accurate tools for making forecasts (like VAR). The purpose of DSGE models is not to generate forecasts. DSGE is for policy experiments and understanding the economy. I don’t see the lack of private sector use of DSGE models as a relevant market test.

13 TH January 10, 2014 at 10:14 am

While I agree, it is not irrelevant. It points to DGSE not being successful for all applications.

14 PJ January 10, 2014 at 11:00 am

Yes, I agree. Well said. No macro model ever can or will be successful for all applications.

15 Dave January 10, 2014 at 11:41 am

Especially when it’s complete BS.

16 Rahul January 10, 2014 at 10:19 am

I’m not familiar with DGSE but how can any model be defended by claiming it’s purpose is not to generate forecasts?

The ability to give correct forecasts seems a key test of the validity of any sort of model. Without correct forecasts what distinguishes a good model from crap?

17 PJ January 10, 2014 at 10:55 am

My view is that DSGE is the best thing we have for running policy experiments. Bayesian VAR models probably generate better forecasts assuming that the future (including policy and reaction to policy) will closely resemble the past.

Rahul raises a fair question about the usefulness of models that don’t generate accurate forecasts. I suspect non-DSGE statistical models do a fine job if the data lie in the fat part of the distribution and aren’t likely to change much.

I am much more confident in a DSGE model for identifying optimal policy under way out of sample circumstances. As an example – what should a central bank do when the federal funds rate is zero and unemployment appears to be well above the natural rate?

18 Rahul January 10, 2014 at 1:31 pm

I don’t know if this is the right analogy but reminds me of Physics: We have very good ways to model, say, the breaking points of metals in tension. Mostly empirical fits. For an applied engineering firm those models suffice & indeed that’s what they use a lot.

OTOH, first-principles’ Quantum Mechanics models do exist but aren’t so good yet and do a worse job then empirical models. Yet we continue to work on them because they have fundamentally more potential when extrapolating to broad circumstances etc.

Is DSGE versus conventional models a similar situation?

19 PJ January 10, 2014 at 1:41 pm

I only know a little bit of Newtonian physics, but that sounds to me like a very good analogy.

20 Adrian Ratnapala January 11, 2014 at 1:46 pm

I doubt that ab initio quantum calculations have any potential to be better predictors than the empirical fits used in these purposes. However, they can sometimes predict real phenomena while also giving people some clue about what the **ck is going on; and that is the whole point of physics.

It would be very surprising if there were no analogous situations in economics.

21 Steko January 10, 2014 at 8:11 pm

“My view is that DSGE is the best thing we have for running policy experiments.”

Why? How is that even possible if it can’t be used to forecast? If DSGE model X says Policy A > Policy B what makes that preferable to German Octopus model Y saying Policy B > Policy A?

22 Rahul January 11, 2014 at 7:55 am

I think it can be used to forecast but just does a way worse job at it currently than conventional options.

The argument defending DSGE runs somewhat like this IMO: (1) There’s reason to believe that at some point DSGE can / will become as good as competition (2) When it does, it will be more robust & more widely applicable and / or it’s inputs / parameters can be more meaningfully set.

Basically a complex black box model that does well when tuned to a specific scenario versus a more general model that tries to capture the phenomenology of the underlying process.

23 Ricardo January 10, 2014 at 12:02 pm

Consider the theory of evolution (specifically, speciation) by descent with modification. Is this a good model? Does it generate (verifiably) correct forecasts?

24 Govco January 10, 2014 at 1:14 pm

From your link:

“Diane Dodd used a laboratory experiment to show how reproductive isolation can evolve in Drosophila pseudoobscura fruit flies after several generations by placing them in different media, starch- and maltose-based media. Dodd’s experiment has been easy for many others to replicate, including with other kinds of fruit flies and foods.”

So, at least, the theory has generated a forecast (this bug will speciate if isolated in maltose) that has been verified and replicated.

25 Manoel Galdino January 10, 2014 at 1:37 pm

If you read some of Richards Dawkins’ books you’ll find some examples of accurate forecasts of evolution theory. There are some lab experiments, ran for about 20 years, that “confirmed” (i.e, results were as predicted) evolution theory.

26 Ricardo January 10, 2014 at 1:49 pm

So…. I’ll repeat my question (which was not meant to have an implicit answer): is this a good model?

On the “yes” side: it generates testable predictions (as noted by Govco and Galdino).

On the “no” side: those predictions are not really very interesting. I want to know how humans will evolve in the next 10,000 years. I don’t really care about fruit flies.

My null hypothesis is that macro models are about as useful as models of evolution. This hypothesis is relevant because I hear lots of people say “macro models are useless” but I rarely hear (informed) people say “models of evolution are useless.”

27 guest January 10, 2014 at 4:29 pm

no one bases public policy on models of evolution.

28 sort_of_knowledgable January 10, 2014 at 6:06 pm

You may not care about fruit flies, but a lot of people care about how other insects might evolve under various conditions such the varying of pesticide usage. For example, there are proposals for malaria control by using pesticides that especially target older mosquitoes. They are the ones that pass malaria because the malaria parasite needs time to grow in the mosquitoes. Since the younger ones still reproduce there is less pressure for the mosquitoes to evolve resistance to those pesticides.

29 Steko January 10, 2014 at 8:16 pm

“I rarely hear (informed) people say “models of evolution are useless.”

People say all the time that models like Intelligent Design are useless. For those keeping score at home, DSGE is Intelligent Design in this analogy.

30 Phill January 11, 2014 at 3:48 am

I understand the point you’re making, and I agree with you.

31 Simone Simonini January 10, 2014 at 10:13 am
32 Noah Smith January 10, 2014 at 3:07 pm

Nope. I only ever post under my own name.

33 CPV January 10, 2014 at 10:22 am

I think of climate modeling and macro as the two most failed attempts at understanding important emergent phenomena, Macro may be the worse of the two as while neither can estimate the magnitude of important variables, macro frequently gets the sign wrong as well. The idea that macro models are useful for understanding policy impacts seems laughable given the debates in the profession.

34 Willitts January 10, 2014 at 11:12 am

When the government shut down last year, one fairly well known economist said that without the various statistics bureaus we were “flying blind.”

Any economist who believes that government is “flying” our economy and frequent course corrections need to be made should not be allowed anywhere near a policy discussion. If the economy were an airplane, government would be the weather or, perhaps, the baggage handlers.

35 1%er January 10, 2014 at 10:23 am

This is completely wrong. This person has clearly not read ANY of the economic research put out by Wall Street over the past decade. Have you ever heard of FRB/US, the Fed’s core macroeconomic model? All of the competent economics shops use it for various purposes and analysis. Jan Hatzius and co at Goldman refer to it biweekly. How else to understand and stress-test the “optimal control” and “forward guidance” framework under which the Fed currently operates? Now of course, this is not at all an endorsement of the forecasting validity of these types of models, but essential to understanding and predicting the reaction function of the central bank. But to claim that the “private sector doesn’t use DSGE” or hire people who understand them is simply incorrect.

36 Jon Rodney January 10, 2014 at 11:05 am

My understanding is that FRB/US is not a DSGE model.

37 John B. Chilton January 10, 2014 at 10:24 am
38 v5 January 10, 2014 at 12:02 pm

I enjoyed this line:
“Saltwater DSGE macro, as best as I can tell, is just a kind of highbrow trolling.”

39 CPV January 10, 2014 at 1:11 pm

Macro modeling is reverse engineered, results driven modeling. Yglesias has that part right. The models are constructed to validate policy preferences. The policy preferences are derived from preferences about the overall size and role of government. The preferences about the overall size and role and government come from individual tastes, preferences, cultural background and mood affiliation. Macro modeling is the opposite of a science.

40 derek January 10, 2014 at 11:00 am

A good friend who is a retired meteorology professor describes to me the methods of measuring accurately how the energy from the sun hits the earth surfaces and how the air temperatures change, how that energy gets mixed into larger atmosphere, and eventually turns out to be weather we experience. Models are built based on all the processes that are quantified, very complex models that are used to predict the weather. From this complexity comes the chaos theory, which is a way of quantifying uncertainty. The models decrease in usefulness as you project into the future, but they work well for short term predictions, with constant adjustments of inputs to realign with reality.

So does DSGE do something similar? I got up this morning, slept in a bed with sheets and a companion, showered, defecated, ate breakfast, fed the dogs, checked Marginal Revolution and other sites. I’ll be leaving for work soon to generate cash to pay for all the things I use. The various materials that go into anything from heating my home to growing the food come from a long process of extraction and processing, transportation manufacturing etc. I would suspect that to construct a model from that basis would take as much work as the meteorological models, and be prone to the same problems with complexity. If an input is mismeasured, or better yet, the range of input that comes from the precision of the means of measurement could influence the output in dramatic ways, then the same chaos theory uncertainty would apply.

With weather, knowing that it is winter and going to be cold is different from knowing that you won’t be able to get out at the airport because of a storm arriving tomorrow midday. Spring is coming in a couple of months, then we will have hot weather, then the fall and winter again. Isn’t that about the level of prediction that macro is good for? I can set up to meet expected demand for my services based on the expected seasons, but not much more.

41 Willitts January 10, 2014 at 11:26 am

The economy consists of trillions of transactions behaving not with the regularity of physical phenomenon but with the peculiarities of the human minds.

A ray of energy from the sun has well defined properties. How millions of people will react to a debt default in Russia does not.

Public utilities can predict and prepare for 50 million toilet bowl flushes during Super Bowl commercials. If financial markets were that predictable, I wouldn’t be at work today.

So we have all these metaphors and analogies that present a plausible but illusory perception of control. The quoted passage cuts through the BS and goes straight to the reality that well worn models do not work well and are not used by people who have a lot of money at stake in predictions (although this fact is disputed by 1%er.

I said this days ago on a different topic: economics and finance would be better off if they banned metaphors and analogies. Some are apt, but most are deceptive.

42 Darren January 10, 2014 at 2:14 pm

I think it would be similar if once in a while, molecules decides to buck the rules of physics and do whatever the hell they want.

43 zz January 10, 2014 at 11:13 am

Having worked in the forecasting industry for awhile, I can tell you why this is: everyone knows that the Lucas critique undermines their models, it’s just that the DSGE models suck, so they use statistical models as a sort of descriptive statistic. Lucas Critique is exactly that, a critique…it doesn’t guarantee the existence at any point in time of a model that accurately reflects the deep parameters of the system, it just tells you that a model that doesn’t do this is fundamentally flawed.

44 Doug January 10, 2014 at 11:33 am

Not that I’m any fan of academic macroeconomics, but maybe we shouldn’t put too much emphasis on the endorsement of macro funds. The strategy has of late been by far the worse performing hedge fund class (link below). The market doesn’t really understand what’s happening with the macroeconomy either. For that matter no one really does, the Austrians, Keynesians, monetarists, MMT, neo-Kenyensians, supply-siders. All of them have been drastically wrong in some way or another in the past decade. Post-2008 we really are clueless when it comes to macroeconomics.

https://www.hedgefundresearch.com/hfrx_reg/index.php?fuse=login&hi

45 Willitts January 10, 2014 at 12:27 pm

The problem isn’t that we didn’t see the crisis coming, but that none of the people who were part of building it had the incentive or courage to stop it.

The problem isn’t that we don’t know what went wrong with the economy. The problem is that the guilty parties don’t want to face the music.

The problem with policy to fix the problem is that there is none. But try telling that to voters.

The prevention is to take away their toys and put them in time out. But that won’t happen either, so get ready for the next one.

46 Dave January 10, 2014 at 11:56 am

This reminds me of a response I read from Mark Thoma years ago that I found so misguided. Here’s his response to someone arguing against the notion of economics being “at last a science” from the Austrian point of view:

Finally, a question for you. If markets work, and Austrians are always right, or nearly so, how come Austrians are such a small group in the profession? All the market incentives are there, there’s no reason at all they couldn’t take over, so why don’t they dominate, or even make up a significant part of the mainstream of economics? Is it the evil government? Or the quality of the goods they are selling?

It’s clear here that Thoma assumes that the ivory tower is the ultimate market test. That struck me as really pompous, to the point that I still remember it 5 years later. Noah is correct to look at industry.

47 Willitts January 10, 2014 at 12:21 pm

He is exactly the economist who said we were “flying blind” during the shutdown.

48 Dave January 10, 2014 at 1:49 pm

Maybe I should’ve been clearer. The point of my comment is that Thoma thought that academia *was* the market test for ideas when it comes to schools of thought for macro. Not its practical application in the real world.

49 dswarts January 10, 2014 at 2:52 pm

I am sure Thoma doesn’t think academia is the market test. He is suggesting that if something (a model) works then academics will explore its usefulness and limits.

50 Dave January 11, 2014 at 4:46 pm

Read the quote I provide. He very clearly is equating academia with the market test.

51 Dave January 10, 2014 at 12:06 pm

One more comment: Even though it has a long way to go, I do see agent based modeling as being a promising alternative to DGSE, although that will require economists who are also talented object oriented programmers (and this will, in fact, be more and more common, I believe).

52 Wonks Anonymous January 10, 2014 at 12:27 pm

I find it plausible that DSGE models are simply not useful for anything, but it’s makes a lot of sense that the private sector doesn’t care much about satisfying the Lucas Critique: private sector actors usually* can’t cause the changes that Lucas warned about.
*Roger Koppl has a theory about “Big Players” who may be an exception.

53 Wonks Anonymous January 10, 2014 at 12:58 pm

Doh, that’s precisely the point Noah was addressing. Shows me for just settling for Yglesias’ summary.

54 paul January 10, 2014 at 1:12 pm

Tyler, excellent question and it seems urgent for macroeconomists to respond. I hope you’ll post on MR any responses worth reading that you come across!

55 prior_approval January 10, 2014 at 1:14 pm

I see ‘DSGE’ and this is what came to mind – ‘The General Directorate for External Security (French: Direction Générale de la Sécurité Extérieure, or DGSE) is France’s external intelligence agency. http://en.wikipedia.org/wiki/Directorate-General_for_External_Security

Just my little mistake.

56 yo January 10, 2014 at 2:10 pm

Yeah I was wondering too, what the fuss about some French secret police. But its record of sinking Greenpeace ships sure beats some ol’ smelly forecasting models. If only they were top models.

57 Scott Cunningham January 10, 2014 at 1:41 pm

Not trying to be a troll, but how far do we want to take this “market test”? I wonder how common it is for private firms to confuse correlation with causation? I’ve been surprised with all the big data discussion, for instance, how that distinction (critical in science) is obscured. So maybe we shouldn’t be impressed with their adoption or lack thereof of any modeling?

58 Silas Barta January 10, 2014 at 3:22 pm

Explain like I’m 5: How do dynamic stochastic general equilibrium models evade the Lucas Critique?

59 OneEyedMan January 14, 2014 at 8:02 am

In standard DGSE models, agents have beliefs about the sources of randomness in the model that match their actual distributions. When they optimize their objectives, they are search for the maximum in the presence of this randomness, so households act rationally in the face of what they expect, and what they expect is true.

60 Dave Barnes January 10, 2014 at 4:19 pm

Is there actually a “market test” for http://www.defense.gouv.fr/english/dgse ????

61 Steve Sailer January 10, 2014 at 7:32 pm

Is it asking too much to spell out acronyms like DSGE?

62 Rahul January 10, 2014 at 11:14 pm

Maybe it’s a good filter. A few posts like this keeps the riff raff away. Otherwise the comment section here would soon resemble your blog.

I like it that TC doesn’t try very hard to cater to the lowest common denominator.

63 The Anti-Gnostic January 11, 2014 at 12:28 pm

Calm down there, tiger. You’re obviously every bit the status-obsessed bigot you accuse others of being.

64 Rahul January 11, 2014 at 2:21 pm

Why is status obsessed bad? I’d rather hear about a macroeconomic intricacy from a professional economist with credentials & training than some random raving redneck with an attitude.

The wave of anti-intellectualism is rather silly. Thinking that simplistic theories based on “common sense” & and vivid analogies can explain the world. Who needs professionals, data & complex techniques when an amusing anecdote can suffice?

65 Euripides January 10, 2014 at 9:34 pm

There is a simple answer: industry generally is trying to get the best forecast. So they favor chartist forecasting models. They don’t really care about understanding how the economy works. So if the price of mutton in Timbuktu is useful for forecasting demand for a product, throw it in the model. Then profit.

DSGEs try to understand how the economy actually works based on fundamental principles like utility maximization and profit maximization in the presence of shocks. If one doesn’t believe that firms and individuals try to do that to an order of approximation, then please take all that stuff out of the microeconomics texts!! There won’t be much left.
BTW, I don’t think I have ever read Tyler’s take on DSGEs?

66 DocMerlin January 11, 2014 at 6:23 am

I did forecasting in space physics and the same applies there. Physics-based models are very poor predictors compared to historical based models (neural nets, regressions, look-up-tables, etc), however the physics based models are useful in learning about the phenomena.

67 Bill N January 12, 2014 at 4:51 pm

What is the industry planning time horizon? and how long is the time lag for the Lucas Critique to have some bite?

68 TallDave January 14, 2014 at 10:39 pm

“I’ve called and emailed everyone I could think of”

I’m not sure what fraction of the world of finance we should assume that covers.

Comments on this entry are closed.

Previous post:

Next post: