How good are RCTs in economics?

K., a loyal MR reader, writes to me:

Doesn't all this RCT stuff strike you as being too easy and atheoretical? Where's the connection between running RCTs and the standard first year grad school toolkit?

I mean beyond the obvious "we have to figure out how to allocate scarce resources". I fear that RCTs will, if not they are already, lead people to basically stop doing economics. I mean look at the development in econometrics that has taken place since the 70s – these were all motivated by the fact that economics must rely on observational data yet claim causality. Although to this point there is the QJE 2004 paper which discusses the improvement to standard error calculations that must be made while using panel data. But its the only paper I can think of which has contributed somewhat to an understanding of economics.

Also, I simply can't shake the feeling that the Ester Duflo school is too much of an institution. Being from India, it feels a little insulting that someone in Cambridge Massachusetts will decide policy in rural India while earning some 6 figure salary in dollars. 

My view on RCTs is simple.  Economists have tried lots of methods, why should we not try to study what actually works and what does not?  It's not as if a) our other methods are all so spectacularly wonderful, or b) RCTs have been shown to lead to some kind of disaster.  Most of all, they are radically under-applied, in large part for reasons of cost and implementation.  Whatever you think of the quality of current RCTs, future RCTs in economics are likely to be better.

The main danger with RCTs is that, in development economics, they will lead to an excess focus on social engineering as a driver of development.  They also will lead people to focus on problems which are amenable to success by piecemeal social engineering, which in turn will lead to biases in our understanding and a neglect of big picture questions about economic growth.  Still, while we should take those problems seriously, they would be the result of the success of the method and any successful method will bring such biases.

Since the RCT method, in economics, requires a lot of $$ to implement and thus it will not spread with the same facility as did experimental economics, which is far cheaper (though still more expensive than econometrics).

RCTs are also one way to see how context-dependent are the results of empirical economics.  If giving lentils works in India but not Uganda, that's worth knowing and how else are we to obtain that knowledge?

Here is Chris Blattman on alternatives to RCTs.  The cited Acemoglu paper is here.


Comments for this post are closed