How good are RCTs in economics?

by on May 9, 2010 at 6:57 am in Economics, Science | Permalink

K., a loyal MR reader, writes to me:

Doesn't all this RCT stuff strike you as being too easy and atheoretical? Where's the connection between running RCTs and the standard first year grad school toolkit?

I mean beyond the obvious "we have to figure out how to allocate scarce resources". I fear that RCTs will, if not they are already, lead people to basically stop doing economics. I mean look at the development in econometrics that has taken place since the 70s – these were all motivated by the fact that economics must rely on observational data yet claim causality. Although to this point there is the QJE 2004 paper which discusses the improvement to standard error calculations that must be made while using panel data. But its the only paper I can think of which has contributed somewhat to an understanding of economics.

Also, I simply can't shake the feeling that the Ester Duflo school is too much of an institution. Being from India, it feels a little insulting that someone in Cambridge Massachusetts will decide policy in rural India while earning some 6 figure salary in dollars. 

My view on RCTs is simple.  Economists have tried lots of methods, why should we not try to study what actually works and what does not?  It's not as if a) our other methods are all so spectacularly wonderful, or b) RCTs have been shown to lead to some kind of disaster.  Most of all, they are radically under-applied, in large part for reasons of cost and implementation.  Whatever you think of the quality of current RCTs, future RCTs in economics are likely to be better.

The main danger with RCTs is that, in development economics, they will lead to an excess focus on social engineering as a driver of development.  They also will lead people to focus on problems which are amenable to success by piecemeal social engineering, which in turn will lead to biases in our understanding and a neglect of big picture questions about economic growth.  Still, while we should take those problems seriously, they would be the result of the success of the method and any successful method will bring such biases.

Since the RCT method, in economics, requires a lot of $$ to implement and thus it will not spread with the same facility as did experimental economics, which is far cheaper (though still more expensive than econometrics).

RCTs are also one way to see how context-dependent are the results of empirical economics.  If giving lentils works in India but not Uganda, that's worth knowing and how else are we to obtain that knowledge?

Here is Chris Blattman on alternatives to RCTs.  The cited Acemoglu paper is here.

anon May 9, 2010 at 7:15 am

I mean look at the development in econometrics that has taken place since the 70s

Most of these developments should have taken place in the Statistics department. It is mind-boggling how much work in statistical methods has been published under discipline-specific names (Econometrics, Epidemiology, Biostatistics, Demography etc.)–this has most likely set back the progress of theoretical and applied statistics by decades.

Don May 9, 2010 at 7:38 am

All those RCT’s in medicine haven’t exactly led folks to stop doing medicine. Why should RCT’s in economics lead folks to stop doing economics?

Mounir Karadja May 9, 2010 at 8:52 am

My biggest problem with RCTs is that we have some of the brighest minds in economics doing basically simple stuff, performing trials with treatments that anyone could come up with. The exception perhaps being treatments inspired by Mullainathan’s behavioral economics view of poverty. Anyway, the “brain-dead” work of RCTs would preferably be handled by NGOs and development agencies, as a matter of efficient division of labor.

I see two big reasons not to be disturbed by the increase in RCT use by top economists, though. The first is that actual randomization now is being properly propagated as an indispensable tool for evaluating aid, something which non-academics would tend to ignore. This is a highly valuable task.

The second is that even if these economists are to some extent allocating too much of their precious human capital to “brain-dead” activities, these guys are still just a tiny fraction of the econ profession. Soon enough, RCTs will become less fashionable, fewer academics will spend time on it, and hopefully much closer to a 100 % of RCTs will be carried out by less skilled people at aid organizations.

Rahul May 9, 2010 at 9:26 am

The feeling I had after watching the Esther Duflo video on TED Talks was something akin to the Emperors New Clothes. Here was the whole world admiring this lady for her genius but I really couldn’t see what was in her talk that was novel or smart or exciting. Yes, some incentives influence people while others do not. What’s new? Where’s the elegance? What has she contributed that we didn’t know before?

anon May 9, 2010 at 9:48 am

Rain, see this Wikipedia page
http://en.wikipedia.org/wiki/RCT

In this case, Randomized Controlled Trial

It also stands for “Rational Choice Theory – a framework for understanding social and economic behavior”

or “Rural Community Transportation, a public bus system in Vermont”

IWantCookieNow May 9, 2010 at 10:58 am

It took me a while to figure out that RCT means “Randomised Controlled Trial” (or something like that).

The Economist has the good habit of explaining even well-known abbrevations when they are used in an article, why not do that too?

LemmusLemmus May 9, 2010 at 11:59 am

I’m a little confused by the charges that RCTs (and econometrics!) contribute little or nothing to “economics”. Maybe if discussants were explicit about what definition of “economics” they’re working with it would all make sense.

Jeff May 9, 2010 at 12:35 pm

I too hate argumentation by saying “that’s not economics”. The standard for something to be economics is that it tells you something about the economy, not that it meshes with the existing economic literature. In fact, the latter is about as unhealthy a definition as you could draw up. Did doctors rule out the use of MRI on the grounds that it wasn’t X-ray?

April Blossom May 9, 2010 at 1:34 pm

My main problem isn’t with the trials but with the people who do them. Almost always, they focus on correcting perceived behavioral “flaws”. This is essentially because they only have intuition about psychology and don’t understand the macroeconomic complementarities that are the key to development. Also typically institutional context is ignored. I strongly believe that the key to development involves addressing these complementarities and designing appropriate (incentive compatible for the key players) rules/institutions. This nudge stuff is garbage. The Kenya fertilizer study is deeply flawed.

k May 9, 2010 at 3:39 pm

I like it how everyone ignores the last sentence in the comment, which is really where this whole thing bites.

And I never said anything about “real economics”, because I certainly don’t know what that it is. I have said and maintain that just running an experiment in itself is unlikely to be very helpful in understanding economics. It is dishonest to distort an argument and make up an implication I never argued for. Where have I said you have to do this or that to be a real economist?

It is PRECISELY that point that one must understand how the world works that makes me have suspicions about the purported usefulness of RCTs. It is all too easy to get funding for a project ostensibly designed to measure an outcome with minimal investment in understanding the people who are to be subjects, or the world they are living in. If you add in the fact that this supposed clean way of figuring out a result is still quite dirty, then the supposed benefits fall even more.

Holding an experiment in the field is very different from a clinical trial of a drug, so that argument doesn’t hold. Still, let us examine this analogy. Medicinal research doesn’t advance because someone decided to test a drug, it advances because people attempt to understand the biology of human beings. This is exactly what I’m arguing for.

Now having done my shouting, let me give an example of what I learned on the field many years ago: I was working in Bombay for a bank, but in its social initiatives group. Yes, these things exist. Now the project I was on involved research on a school for kids living in some slums in Bombay. The leader of that school and I had a conversation once about midday meals – the giving of lentils for example. She was completely against the policy because she said that a packed lunch is a very important link between the mother and her child (it is mostly mothers who prepare lunches in these communities), and depriving either the parent or the child of this has the potential to cause a break in a family tie.

Further, the teachers of the school would inspect the lunches brought and if it turned out that some children were consistently bringing in low-nutrition food then the teacher would try to teach the child or the parent what is nutritious and what is not. So, you see, there is more to this midday meal scheme than meets the eye.

What the above points to is just the fact that the real world is complex, that problems of development and their solutions are intricate and must come from the bottom up – all of this is ignored when you claim to run an experiment that identifies some outcome. It isn’t like I’m talking completely out of thin air.

k May 9, 2010 at 11:40 pm

I don’t see the point in debating what is a real economist or not.

We have to understand what RCTs do. My view is, I repeat, it makes false promises of generating a clean result. So far, seeing what the people I believe who know what they’re talking about say about RCTs, this is at least a debatable point.

I never said this tool was inherently worse or better than any other (unlike some proponents of the technique) only what it was leading people to do.

Steve May 10, 2010 at 1:39 am

I think the reason a lot of people don’t like Duflo’s work is that RCTs seem “too easy.” She is “just” running experiments.

But it’s worth remembering

(1) she has written a number of more traditional (and very high quality) empirical papers (Labor market consequences of school construction in Indonesia in particular).

(2) It’s hard to find a good interesting topic for an RCT and do the econometrics for it right, design the experiment right, etc. It’s not like the experiments are so clean and simple she just does a t-test at the end of the day.

No one noticed an error in how the standard errors were bootstraped in the Miracle of Microfinance paper because is was so technical until David Roodman pointed it out (who has training as a theoretical mathematician and is famous for his Stata programs).

If you’ve ever met Prof. Duflo you can tell she is a very very sharp thinker in terms of econometrics and microeconomics. The prize was well deserved. The area is obviously well deserving of research, whether by “economists” or … well whatever they would be called.

DG Lesvic May 10, 2010 at 3:01 am

Same as above, but slightly amended, with apologies.

“How good are RCTs in economics?”

“Randomized control trials,† “field experiments,† and “experimental games† are just that, fun and games for idle academicians. There can be no actual controlled laboratory experiments in the field of human action, for you cannot control actual human beings. You can only control the imaginary ones in your mind. And its imaginary constructions of a universe with but one factor of change are the only “laboratories† of economics.

Since no one has ever been able to explain how you observed the Invisible Hand, and, if you couldn’t observe it, measure, count, or calculate it, economics is neither empirical nor mathematical.

Since it is a science not of ends but means, it is not psychological but practical.

Since it is a science not of particular times and places, but all times and places, it is not sociological but universal and eternal, and not predictive but explanatory.

It is not an empirical, mathematical, psychological, sociological, nor predictive science, but a theoretical and “logical,” or, more precisely, praxeological, science, a science of action in the market, and not of any particular times and places but all times and places.

r4 dsi May 10, 2010 at 6:56 am

Very well explained.I always believe RCT is a convincing theory of human behavior at all times and under all circumstances.It provides a reasonable explanation of choices in the vast majority of cases of ordinary behavior, and ordinary behavior in market and non-market settings arguably constitutes the bulk of human existence.

liberalarts May 10, 2010 at 7:22 am

I think that there is a lot of jealousy here and elsewhere about Duflo’s success with RCT. I am sure that she is quite conversant in all of the high tech methods, or she couldn’t have survived MIT’s Ph.D. program. The point is that RCTs were not being used to drive economic development and poverty thinking. She could thus swoop in and make great progress without having to use her high tech tool box. To argue that economists are too smart to enter and improve unpopulated, low tech areas of research is just silly.

Ricardo May 10, 2010 at 8:22 am

Let’s get specific and refer to a specific paper: Banerjee, Duflo, Glennerster and Kinnan (2009) [the Hyderabad microfinance paper]. I think it is quite clear what this experiment does and does not say. It does not say microfinance does not have some long-term impact on standard of living. It does not say the impact has to be the same for all individuals. It does not say the impact is the same in Hyderabad or Dhaka or Delhi or Hanoi or Lima.

What it does say, though, is that people who are claiming the key to development is relaxing credit constraints are doing so without adequate evidence. That is an important lesson. It means we should not pin too many hopes on microfinance — especially on institutions that operate on the scale of Spandana and that naturally have to be selective about who or where they serve. That strikes me as a non-trivial contribution to knowledge (e.g. it has implications for the Hernando de Soto view of poverty as lack of property rights as well). So where’s the problem?

Jordan May 10, 2010 at 10:16 am

“The fact that we are having this debate must signify something is the problem – yes?”

Not necessarily. While I agree that this particular debate is worth having, as a general rule “disagreement = evidence of a flawed understanding” doesn’t hold; for example, take Creationists.

“I have stated my opinion now multiple times as clearly as possible. Please do not twist it to some extreme and then attack the extreme.”

I’ll let other commentators’ posts speak for themselves, but in my case most of my objections were directed largely towards other people’s comments – for example the post immediately preceding mine, as I noted – not towards your original email.

I did want to address the salary issue specifically since you had more or less invited everyone to do so and yet I felt it was the weakest point in your argument. I’ll admit your original description of RCTs as “too easy and atheoretical” set off a couple of alarm bells, as it does sound a little like “harder math” = “better economics”; but other people seemed more flagrant in advancing this view, and obviously to judge by your follow ups its not what you meant to convey.

Also, this:

“I fear that RCTs will, if not they are already, lead people to basically stop doing economics”

is clearly hyperbole. I’d be deeply shocked if we see any reduction in the number of non-RCT economics papers published, even if say Professor Duflo goes on to win the Nobel.

“Then, do you really need an experiment to understand nuances? It is not like the insight offered to me by that particular leader of the school took me decades of effort and work – just the acknowledgment that local people are likely to understand their problems better.”

Yes, people always have access to a whole heap of special information and insight into their own problems that is critically relevant to outsiders who want to help – not to mention that helping people help themselves is the most empowering and dignifying way to give aid.

However, your story about the lunches is still fundamentally anecdotal. The leader of the school has identified a plausible reason to believe educating families about nutrition might be better for school kids than simply providing food, and that’s a great realization that outsiders likely missed. You don’t actually know which approach is superior, though.

Say someone were to run a RCT in Uganda, with free school lunches in some villages and free nutrition education for families in others, and compared the outcomes. Obviously, they would need to be clever and careful in the experimental design and interpretation of the results, lest they miss the nuances; for example, measuring nutritional outcomes alone would probably be a bad idea, you’d also want to see what the effects on the mother/child relationship are. Assuming the experiment was done well, we’d have a better understanding of the issue.

Sure, the Ugandan kids in this scenario are being “experimented on” – in the sense one group is getting free food and the other free parental education, with the intention of finding out which group gains the most. Just like subjects in a clinical trial of some new anti-depressant vs prozac are being “experimented on.” I really can’t see what the problem is with either scenario, though.

Of course, the experiment and many others like it could be done poorly – precisely because this kind of work is neither as easy nor as dull as some here seem to think. No doubt there will be many weak RCT papers in economics published, just as there are weak papers published in all sorts of fields using all sorts of methodologies. Nonetheless, time will weed out the bad results and leave us with the replicable ones, and we will have a net advancement in our understanding of the world.

DG Lesvic May 10, 2010 at 3:53 pm

I need to amend my own statement about the difference between economics and sociology. The real difference is that economics is a science not of changing mores but unchanging laws.

As for Jordan, perhaps I do him an injustice, but all I see from him are pronouncements with nothing backing them up. I can’t argue with the pronouncements themselves, only the reasons for them. And I don’t see any.

DG Lesvic May 11, 2010 at 2:18 pm

Tracy,

You wrote,

“We can observe what might be the effects of…the invisible hand.”

I have written, elsewhere,

“But…observing its effects is not observing the Hand itself, and is not economics. Observing an economic event doesn’t make you Adam Smith any more than seeing an apple fall from a tree made you Sir Isaac Newton. Economics is not what any man in the street could observe, but the reasoning no one could.

Without it, empirical data is a meaningless jumble. There is economics without the jumble, but not without the reasoning, and, without the Chicago School altogether, but not without the Austrian.

Nobody is against historical evidence, just its confusion with economics. It may augment but not supplant it, and, when contrary to it, is not historical but anecdotal.”

See http://econotrashtalk.org/#Observing_the_Invisible_Hand:

You wrote,

“And as for your claim that the Invisible Hand can’t be calculated, this is just false. The Invisible Hand has already been calculated, numerous times.”

What was the conclusion of those calculations, the final numbers attributed to the Invisible Hand, and true of it for all times and places?

You wrote,

“But economics is a science of both ends and means.”

Economics is a science not of the motivations for action but of action itself, asking not why men aim at certain ends but whether their actions lead to them or not.

The fact that people who call themselves economists digress from economics doesn’t make their digressions economics. Psychology is one thing and economics another.

My first explanation of that difference, above, was wrong, but note that I amended it, as follows:

“I need to amend my own statement about the difference between economics and sociology. The real difference is that economics is a science not of changing mores but unchanging laws.”

As you said, economic analyses frequently refer to the events of particular times and places. And though that is an expedient way of presenting economics, it doesn’t change that fact that it is a theory of all times and places and not just of some.

“Economics†¦adopts for the organized presentation of its results a form in which aprioristic theory and the interpretation of historical phenomena are intertwined†¦this†¦procedure†¦has given proof of its expediency. However†¦uncritical and superficial minds have again and again been led astray by careless confusion of the two epistemologically different methods implied.†

Mises, Human Action, 3rd Ed., P 66

DG Lesvic May 11, 2010 at 6:39 pm

Tracy,

Haven’t had a chance yet to get into all of your excellent if not exactly correct points, but your final paragraph above got it just about right. I am trying to distinguish between economics proper and the bastardizations of it.

TracyW May 12, 2010 at 3:59 am

DG Lesvic: Was observing an effect of physics, such as an apple falling from a tree, itself physics?

If the observing is being done by someone thinking about some laws of physics, yes. Be that trying to originate them, or trying to test existing theories. Whether dogs or cats think about the laws of physics is outside my knowledge.

All that observation of an empirical event can tell us is what happened at a particular time and place, but not what will happen, nor even tend to happen, at all times and places, and is not economic theory but simply history.

I agree that observation of an empricial event is, by itself, not economic theory. I disagree that it is simply history. I think the observation of an empirical event can easily be part of economics.

The point was that you could not derive what never changes, the laws of economics, from what constantly changes, the data of economics.

I disagree that the laws of economics never change. If you wish to define a law of economics as something that doesn’t change, then I will agree with you that the laws of economics never change, but I will then simply state that economics includes then both your unchanging laws of economics, and also matters that are more dependent on space and time.

I said that a conclusion was a law of economics only if it was true for all times and places, and that there are no data that are true for all times and places.

My apologies for misreading you. I thought you were claiming that economics as a whole is neither empirical nor mathematical, because the Invisible Hand is not directly observable. Now you are saying you were talking not about economics as a whole, but only about laws of economics, which you define as things that are true for all times and places. Is my understanding now right?

But isn’t there still a Theory of Money, apart from the history of it, and isn’t it, apart from the history, a useful study?

Perhaps. But studying the history of money is also part of economics.

I clicked on your link in your first post and got taken to a long book which appeared to be about the American Jewish experience and capitalism, being neither American nor Jewish this doesn’t interest me. (No disrespect to American Jewish economic thinking, but I only have so much time.) Somehow I suspect I would have to do an awful lot of wading through extraneous material to find out that all you’re doing is making some assertions I disagree with.

Comments on this entry are closed.

Previous post:

Next post: