How random are RCTs?

From Hunt Allcott and Sendhil Mullainathan, “External Validity and Partner Selection Bias”:

Program evaluation often involves generalizing internally-valid site-specific estimates to a different target population. While many analyses have tested the assumptions required for internal validity (e.g. LaLonde 1986), there has been little empirical assessment of external validity in any context, because identical treatments are rarely evaluated in multiple sites. This paper examines a remarkable series of 14 energy conservation field experiments run by a company called Opower, involving 550,000 households in different cities across the U.S. We first show that the site-specific treatment effect heterogeneity is both statistically and economically significant. We then show that Opower partners are selected on partner-level characteristics that are correlated with the treatment effect. This “partner selection bias” implies that replications with additional partners have not given an unbiased estimate of the distribution of treatment effects in non-partner sites. We augment these results in a different context by showing that partner microfinance institutions (MFIs) that carry out randomized experiments are selected on observable characteristics from the global pool of MFIs. Finally, we propose two simple suggestive tests of external validity that can be used in the absence of data from many sites: comparison of observable sample and target site characteristics and an F-test of heterogeneous treatment effects across “sub-sites” within a site.

The gated paper is here, ungated here.  For the pointer I thank @JustinSandefur.


whats an RCT ?

'RCT: randomized controlled trial'

Or 'A randomized controlled trial (RCT) (or randomized comparative trial) is a
specific type of scientific experiment, and the preferred design for a clinical trial.'

Well that clears things up. I was wondering what all this had to do with really crispy tacos.

from conclusion: "Third, some have argued that "mechanisms" can in some circumstances be more easily generalized across sites and domains than average treatment effects for specific projects (Deaton 2010a, Deaton 2010b, Ludwig, Kling, and Mullainathan 2011). If this is the case, researchers can focus on identifying mechanisms upon which policy decisions hinge and designing empirical studies to tease them out." Yeah, a careful paper about inference...what kind of conclusions can we reasonably draw and under what conditions. My (mood-affiliated) take away is that there is no single gold-standard, multi-purpose method for policy evaluation. As the authors stress this is not an anti-RTC paper, but rather they shine a light on the effective limits of RTC...and some limits may require other methods to address. Yet, another example that economics is (and always will be) a social science. People are not random and unobservables are often a big deal.

I was in a clinical trial in 1997 for the breakthrough monoclonal antibody Rituxan, which since became the world's top selling cancer drug, which is probably why I'm alive today. It wasn't called a randomized trial and I was assured I wasn't getting a placebo, but I was still worried that I'd get a placebo. Within five minutes of getting the first treatment, I got cold shakes that went on for four hours. This was extremely reassuring in that I could tell I was getting strong medicine. If I had had no reaction to the medicine, I might have dropped out and looked for some other treatment.

As far as I can tell, Tyler's headline is extremely misleading. It seems to have mislead Steve, whose comment has absolutely nothing to do with the paper, but matches Tyler's headline.

Claudia's quote is much clearer than Tyler's. This paper is not about fraudulent trials, but about people who get things to work actually proving it; and people whose systems fail hiding in the shadows of those whose similar-sounding systems do.

Gold Price Chart so beautiful that it will

make you want to lick the screen:
Max Goldberg, Founder.

Steve, I am totally agree with your thoughts. Keep doing these type of work.

Comments for this post are closed