Evaluating the Millennium Villages

Michael Clemens writes:

Here I discuss a new research paper that I wrote with Gabriel Demombynes of the World Bank. We ask when it’s important to take great care in measuring a project’s impacts, and we illustrate one concrete case: the Millennium Village Project. We show how easy it can be to get the wrong idea about the project’s impacts when careful, scientific impact evaluation methods are not used. And we detail how the impact evaluation could be done better, at low cost.

There are some good graphs at the first link and here is the associated podcast.


Rigorous, yes
Analytical, yes
Quantitative, yes
Scientific, no

It is becoming increasingly important to define and use the term science precisely.

There are a many cases, this being one, in which a rigorous, analytical review of facts would be incredibly valuable, but science as such brings nothing to the table.

I have seen scientists show up in development projects (and other fields in which parameters are a bit softer than they are used to) and apply rigorous lab methodology to reach complex conclusions.

But what you find out at the end of their work is that in order to make the evidence align with their rigorous and inflexible methodology, they have to make very simple and usually inaccurate assumptions.

I would expect that rigorous, quantitative economic analysis would bring much more to this than scientific analysis.

I am, by the the, an enormous promoter of science and do think it can provide valuable guidance in a range of areas from policy to morality. Its methodology has influenced and transformed other fields.

But let's get this straight. The hierarchy looks something like this:

Human liberty allows humans to decide on issues rationally. Rationality depends on analysis. Science is one of the most power forms of analysis known to mankind and one that we need to increasingly seek guidance from.

Science has diminishing returns as it's applications extend beyond its base. Science is perfect for doing science, scientific methods have transformed analysis in non-scientific fields, science just gets in the way and you get into areas that are inherently non-scientific

But that does not mean we should abandon all non-scientific analysis.

Those who seek to promote rigorous, quantitative analysis in the broadest application possible, need to love science, but to restrict it to areas where it is applicable.

Alternately, we could just redefine the word science to mean "rigorous quantitative analysis", but then economists would be able to say they are doing science and I suspect all of us here know how hysterical scientists get at the mere idea that economics could be a science.

So, I think we should just understand that science and analysis are not interchangeable terms. Love science, but don't treat it like a creed.

I am personally convinced that investment is the best aid.

Any village which gets about the same amount of money but uses it for investment into profit-oriented businesses will see bigger local increases in a wider area of metrics than other villages over a 5, 10, and especially 15 year horizon.

Profit is the best one-number measure of sustainability. Maximizing it, so far, is the best way to organize humans peaceful to create wealth, and the benefits of that wealth created.

I'd like to see a completely unfunded (aside from salaries and travel costs of the experts) project. I think the only real success will be one that produces money instead of costing money.

If experts can only show progress by pumping money, they haven't really shown anything other than the ability to (raise and ) pump money. Cost effectiveness experiments are great, but until they can show endogenous growth, how can we know that the experts are capable of building with local, sustainable knowledge?

Comments for this post are closed