*Experimental Conversations*

The editor of this truly excellent book is Timothy N. Ogden, the subtitle is Perspectives on Randomized Trials in Development Economics, and the contributors include Angus Deaton, Dean Karlan, Lant Pritchett, David McKenzie, Judy Gueron, Rachel Glennerster, Chris Blattman, and yours truly, with a focus on randomized control trials and other experiment-related methods.  Here is one bit from the interview with me:

I would say that just about every reputable RCT has shifted my priors.  Literally every one.  That’s what’s wonderful about them, but it’s also the trick.  You might ask, “why do they shift your priors?”  They shift your priors because on the questions that are chosen, and ones that ought to be chosen, theory doesn’t tell us so much.  “How good is microcredit?” or “What’s the elasticity of demand for mosquito nets?”  Because theory doesn’t tell you much about questions like that, of course an RCT should shift your priors.  But at the same time, because theory hasn’t told you much, you don’t know how generalizable the results of those studies are.  So each one should shift your priors, and that’s the great strength and weakness of the method.

Now, you asked if any of the results surprised me.  I think the same reasoning applies.  No, none of them have surprised me because I saw the main RCT topics to date as not resolvable by theory.  So they’ve altered my priors but in a sense that can’t shake you up that much.  If you offer a mother a bag of lentils to bring her child in to be vaccinated, how much will that help?  Turns out, at least in one part of India, that helps a lot.  I believe that result.  But 10 years ago did I really think that if you offered a mother in some parts of India a bag of lentils to induce them to bring in their kids to vaccination that it wouldn’t work so well?  Of course not.  So in that sense, I’m never really surprised.

And this:

One of my worries is RCTs that surprise some people.  Take the RAND study from the 1970s that healthcare doesn’t actually make people much healthier.  You replicate that, more or less, in the recent Oregon Medicaid study.  When you have something that surprises people, they often don’t want to listen to it.  So it gets dismissed.  It seems to me that’s quite wrong.  We ought to work much more carefully on the cases where RCTs are surprising many of us, but we don’t want to do that.  So we kind of go RCT-lite.  We’re willing to soak up whatever we learn about mothers and lentils and vaccinations, but when it comes to our core being under attack, we get defensive.

I very much recommend the book, which you can purchase here.  Interviews are so often so much better than just letting everyone be a blowhard, and Ogden did a great job.


Got any links to the Rand and Oregon studies? I'd like to look at those.


Thanks! Great link! Lots of stuff there.

Am I correct in taking the second quote as essentially expressing a preference for frequentism over Bayesianism, at least when it comes to RCTs? The sentiment is "disregard your bias when considering results", which seems directly contra Bayes. (No shade intended to frequentism, of course; just looking for second opinions on that exegesis)

Additional concern: how much work is "reputable" doing in "every reputable RCT has shifted my priors"? It seems like if I like a study, I call it reputable and it shifts my priors; if I don't, I call it disreputable and it doesn't shift my priors. This is probably just me being overly cynical, but my cynicism has a good track record so far, and I don't think that sentence necessarily says much.

A Bayesian always "updates their prior" when receiving any evidence.

"Disregard your bias when considering results" usually means "use a broadly distributed prior probability distribution" in the Bayesian context. I suppose a literal reading does seem a bit Frequentist, and that is what Fisher was after.

But there should be limits to such broad mindedness. For example, a published study comes along that shows women flip their votes (from Dem to Rep or from incumbent to challenger) as a function of their menstrual cycle to the tune of 25% of the time, p < .05. Should you be open minded to that evidence? No, not very, at least if you are aware that a huge amount of research shows that nearly no one flips their vote in the last few weeks before an election. The prior likelihood of such a large effect is therefore nearly zero, and one study (even with p<.05) is not compelling. (and after all, you get p < .05 on 1 in 20 samples even when there is no real effect.)

I think you misrepresent the results of the Oregon Medicaid experiment a bit. The study simply didn't have that much power. The point estimates of the effects for pretty much all the health indicators had the expected *signs*, but the standard errors were large, so you couldn't reject the null of no effect.

A major reason for the low power, despite having a seemlingly large sample, was that the randomization instrument turned out to be fairly weak: many people in the treatment group didn't sign up for Medicaid, and many people in the control group ended up getting Medicaid anyway. So the "intent to treat" effects (what we actually measure) were small, and the estimated "local average treatment effects" were larger but with big standard errors. (For some reason, I never saw any discussion of whether the point estimates for LATE were small or large in clinical terms, relative to what one might hope for.)

Even Casey Mulligan was critical of the way the study was commonly interpreted, despite his political leanings:

When an underpowered study replicates "more or less," it at least provides some evidence that the RAND study wasn't dramatically wrong. That's what I take the "more or less" to mean.

Even more interesting was Mulligan's column on why he was against Obamacare, arguing that it was going to reduce the labor market as it would be more appealing to stay or go on unemployment and gain healthcare benefits: https://economix.blogs.nytimes.com/2013/03/06/health-reform-the-reward-to-work-and-massachusetts/?_r=0 I believe he has been proven wrong on that one.

That would reflect in the labor force participation rate rather than the unemployment rate. He may have been correct.

I think Tyler's point at the time was that there is just as much evidence that Medicaid coverage increases cigarette consumption as it reduces diabetes related conditions.

I think what was surprising about the Oregon study wasn't that it proved one thing strongly one way or the other, but that it didn't.

The same with development economics. One would think that doing something like providing water, for example, or some other basic amenity would have a large effect, but the studies find that they generally don't, or not to the extent that would be expected.

In other words, the priors are often very very shallow and display little understanding of the complexities of people's lives and the very difficult to understand series of influences that drive people's actions. Someone was doing something for child health in Afghanistan, often very low hanging fruit with quick results, but the reaction from the locals was how are we going to feed these children?


Econtalk with Chris Blattman where he describes a study that he set up and how it changed, wrong word, expanded his understanding of the challenges of development economics.

I think often it isn't someone changing their priors as simply going from ignorance to a start of understanding. Ignorance not as a pejorative, but as a description.

Being right for the wrong reason is often worse than being wrong. Is health care overrated? Consider "survival rates" of cancer patients in America (where patients get lots of health care) and "survival rates" of cancer patients in places that don't get as much health care. Conventional wisdom (and the AMA) would lead one to believe that "survival rates" in America are vastly superior to those in other places. But they aren't: it depends on the meaning of "is", or in this case "survival rate". Survival rate is measured from the time the cancer is discovered until death. In America, with all that health care, cancer is detected much earlier than in other places; hence, "survival rates" are much longer in America, even though cancer patients don't live any (or much) longer. Now consider my sister, who died many years ago of cervical cancer. When she was first diagnosed, I set about finding the best oncologist with a sub-specialty in cervical cancer. To my surprise, he was located in the low country, not far from where my sister resided. I was curious why such a renowned oncologist would be located in the low country (rather than, for example, in New York) and was told that he went where the patients are: there is an extraordinarily high incidence of cervical cancer in the low country. Why is that? I concluded that it must be the result of all those chemical plants in the low country, dumping their toxic waste in the rivers and streams and in the air; for those not familiar with the low country, chemical plants are ubiquitous. I was right about the high incidence of cervical cancer in the low country but for the wrong reason: the majority of the low country population is poor and historically received no routine health care, such as the periodic pelvic exams women in affluent areas expect annually. It was the absence of routine health care in the low country that resulted in the high incidence of cervical cancer. In sum, people who receive little or no health care would be greatly benefited from routine health care, while those who receive lots of health care would be only marginally benefited.

Just to be clear, are you saying that routine care would prevent cervical cancer, or just that routine care would find it sooner, and then... what exactly? Because you say that death rates are the same whether it is found early or not.


"has shifted my priors": is that a rather evasive way of saying "has falsified my prejudices"? After all, we're not really chatting about sitting down and doing a Bayesian calculation, are we?

You think every single RCT test falsifies his prejudices?

Dunno. Depends on how inaccurate his prejudices are, I suppose. What do you think he meant? Why do you think he phrased it as if he's some twenty year old who's just learned the jargon?

Comments for this post are closed