First independent look at the Millennium Village project

by on November 28, 2011 at 7:34 pm in Data Source, Economics | Permalink

You will find it summarized here, excerpt:

Working on her own, without the collaboration or endorsement of the MVP, Kenyan economist Bernadette Wanjala of Tilburg University collected data on households in or near the site at Sauri, Kenya, where the project was launched in 2005. She interviewed 236 randomly-selected households that had been exposed to the MVP’s large package of agriculture projects, education programs, infrastructure improvements, and health/sanitation works. She also interviewed 175 randomly-selected households from an area of the same district (called Gem) that was not exposed to the intervention. She wanted to compare the two groups to see for herself whether or not the project had done what it promised: to lift the treated households out of poverty in a few years’ time and spark “self-sustaining economic growth”.

In their just-released paper, Wanjala and her colleague Roldan Muradian of Radboud University use the new survey data to measure the project’s impact on poverty. They carefully compare treated and untreated households that were otherwise similar in many ways—such as household composition, adults’ education, fertility, economic sector, and land holdings. Because this project is large and intensive, spending on the order of 100% of local income per capita, it is reasonable to hope that it might substantially raise recipients’ incomes, at least in the short term.

Wanjala and Muradian find that the project had no significant impact on recipients’ incomes.

How is this possible? While Wanjala and Muradian find that the project caused a 70% increase in agricultural productivity among the treated households, tending to increase household income, it also caused less diversification of household economic activity into profitable non-farm employment, tending to decrease household income. These countervailing effects are precisely what one might expect from a large and intensive subsidy to agricultural activity. On balance, households that received this large and intensive intervention have no more income today than households that did not receive the intervention.

I would gladly publish or link to a response from Sachs or others at MVP.

1 Brett Keller November 28, 2011 at 7:37 pm

Looking forward to reading the study, but your title is misleading — randomly selecting the households to survey and doing the same with a control group might be the best possible design, but “RCT” is generally reserved for studies where the treatment itself is randomly assigned. That’s not the case here because the MVP was (unfortunately) not designed to allow for the best possible evaluation. Watch Sachs et all argue the study is flawed because of that, too.

2 Tyler Cowen November 28, 2011 at 8:11 pm

thanks, corrected…

3 Pragmi November 28, 2011 at 8:09 pm

Wish Easterly’s blog was still up. Oh well. This is sad if not at all surprising.

Also, I liked the 90s policy adviser version of Sachs better than the Bono version. The former discussed complex economic policy reforms with Goni. The latter chants about the 1% with OWS.

4 maguro November 28, 2011 at 8:36 pm

Liberal do-gooderism doesn’t actually work? Shocking!

5 Jamie_NYC November 28, 2011 at 9:18 pm

It does. The subsidies were too small!

6 Brett November 28, 2011 at 10:23 pm

That’s exactly what Sachs will say, along with disputing some of the methodology. It’s what he always says when his methods are called into question.

7 JWatts November 29, 2011 at 10:34 am

The project was doomed to failure due to the low subsidies. We argued for subsidies at least twice the size of the ones we got. However, without the subsidies the household income of the villagers in question would have plummeted 50%. So in reality the subsidies doubled the amount of income the villagers had, but we need to double the subsidies again to get the villagers out of their AD quagmire.

😉

8 Sohier November 29, 2011 at 12:45 pm

More like projects based on a romanticised view of the role of small scale farming in the modern economy will fail. Shocking!

9 Rahul November 29, 2011 at 2:00 pm

Exactly! If MVP had instead had tried to, say, set up a smoke belching industrial plant it’d have been so non-cool.

10 k November 28, 2011 at 10:53 pm

So it made people re-allocate their labour, why is this necessarily bad or, not good?

If farming has a long run payoff, better as compared to non-farming, then surely that is making people better off.

It might simply be for instance that men get to stay at home more, not earning less, but that in itself could be valuable. Income will not measure this.

11 Cliff November 29, 2011 at 12:04 am

The stated goal is to increase income.

12 dan1111 November 29, 2011 at 5:44 am

In addition, the project spent “on the order of 100% of local income per capita.” It has to produce much more than a small benefit to be considered a success.

13 k November 29, 2011 at 8:05 am

yes, sure, but maybe that’s wrong.

14 Rahul Shastri November 29, 2011 at 9:02 am

The aim of the project is to achieve the eight millennium development goals, income increase is just a small part of the first MDG.
The paper here does not reflect on the success or failure of the project, only its possible failure in achieving one of the sub-goals.
The problem of the villages self sustaining the project in the future is linked to incomes yes, but we do not know the baseline at which we started: it is possible the villages started at hunger, are now moving to food security and later may evolve to diversification with small scale industry.

15 NK November 30, 2011 at 1:22 am

It doesn’t matter.
The money used didn’t come out of nowhere.

16 TheCrankyProfessor November 28, 2011 at 10:54 pm

Techno-wishism collapses again. Professional economists will never learn to stop underestimating the power of human nature to evade their steely gaze.

17 Bill November 28, 2011 at 11:14 pm

I don’t reach the same conclusion as Tyler. Go to p 25 of the report. Increased ag production was consumed in the household compared to the control. The report says that cereal storage facilities, which would permit farmers to sell when the market is not depressed, were still being constructed, for example.

This is more nuanced and should be read by yourself. I would be interested in what ag economist or rural dev folks say about this.

18 Jeremy November 29, 2011 at 4:13 am

This is an important point; one wonders whether the nutritional status, health or productivity of the MVP households were enhanced, even if theere incomes were not.

19 joshua November 29, 2011 at 6:34 am

Shhhh! Stagnant wages = TGS, remember?

20 Brett Keller November 29, 2011 at 11:48 am

Jeremy — you’re right that that’s important, but it’s a somewhat separate issue as one of the major claims of the MVP is that it will (and has) increased incomes, not just nutrition/health outcomes. (I think the MVP evaluations so far have largely focused on health outcomes, for instance, but it would be nice if there were more rigorous evaluations of those as well.)

21 ad*m November 29, 2011 at 10:56 am

Bill, I am sorry. On page 25 it does not say that. It does say on page 29:
“However, very few of the cereal banks that were created are currently operational and successful, due to lack of proper management and consequently lack of trust among cereal bank members.”

Bolding mine. My take-away is that there seems to be some setpoint for income, or equilibrium-of-poverty if you want to call it that and that if a source of income is increased, another source is decreased by the same amount.

And generally, low trust leads to poverty. But we can see that In Detroit.

22 Bill November 29, 2011 at 1:34 pm

Ad*m,
1) The estimate of effects begins on p. 25

2) The full quote shows that the VERY FEW of the cereal stores that were to be created (so that wealth from selling across time could increase): “Thirdly, in Sauri the majority of households sell their produce immediately after harvest, when prices are low. Thus, they realize very low income gains. It is for this reason that the MVP introduced cereal banks, which were meant to cushion farmers against selling their produce at the time when prices are depressed. However, very few of the cereal banks that were created are currently operational and successful, due to lack of proper management and consequently lack of trust among cereal bank members.”

3) What this study really shows is a large increase in farm productivity constrained by small farm size AND the failure to develop a market for the surplus, which then gets consumed on the farm. One of my neighbors is a retired Ag Economists, who goes to the god awful places in the world, often contracting diseases. Under AID contracts, he basically teaches teachers on ag improvement. BUT IT DOESN’T stop there: He often talks about how you have to develop the markets for what is produced to be successful and how what is produced may have to change. For example, re recently did some work in South America visiting an area that was very suitable for dairy, and there could have been higher dairy production had certain ag practices changed or been improved. But, he didn’t recommend dairy: instead he recommended goats milk–milk that he found could be transported to the US or condensed in that country, whereas dairy was a losing proposition. So, what they did was work on a milk processing facility that could do both dairy and goats milk! The farmers are now figuring out that goats milk is a winner.

In this case the authors pointed out that that the cereal storage facilities had not been developed, and those that were, were IN THEIR OPINION and without support, were successful. Go back and parse that short sentence again: few created, those that were operational “few” were successful. Now, go back and ask: did the study tell you how many were planned? How many were operational? How long they had been in operation? What were the criteria they had for suceess? And how did they determine there was a lack of trust? Did the paper tell you about “MARKETS”–in other words, linkage to the market between the storage facility and a market? No to all counts,.

23 Bill November 29, 2011 at 6:06 pm

exchange successful for unsuccessful.

24 Cliff November 29, 2011 at 6:13 pm

Bill,

“Go back and parse that short sentence again: few created, those that were operational “few” were successful.”

You have that completely wrong. You need to go back and parse it again. Very few OF THOSE CREATED are both operational and successful. That implies many were created, but very few successful.

25 Bill November 29, 2011 at 7:28 pm

Cliff, I can read the sentence two ways. Given that there was a 70% increase in productivity and the reference to low prices and home consumption of surplus, my reading of the sentence on cereal storage makes the most sense. However, given the role that cereal storage–and moreover, market access makes–I am surprised that the authors did not amplify or make their sentence clearer. But, here are the quotes from the report that further support the view of ag surplus and selling at low prices and thus home consumption. While talking of home consumption, I would also want to know if this is a default definition: ie, if it is not sold, it is presumed to be consumed at home. What “at home” means in a tribal context may change: if you have a surplus, not only do you consume, but you share with more distant relatives. Again, some of the results–the large improvement of agriculture productivity–does not match unless you were to find that there were no markets or market access for the increased farm output.

“The average level of agricultural productivity in MVP is almost twice (25 bags per hectare) as much the productivity in non-MVP households (14 bags per hectare). The income derived from agricultural surplus is also significantly higher in MVP than in non-MVP……In order to assess whether the differences in agricultural productivity and income can be attributed to the implementation of the MVP, we conducted propensity score matching……From the results, it can be deduced that there was a statistically significant increase in agricultural productivity by an average of 10.1 bags per acre of land (about a 70 percent increase in productivity) as an effect of the MVP. The increase in productivity can be attributed to the use of improved production technology, namely the use of improved seeds and fertilizer.(26)…An increase in income was expected to arise from the interventions due to increased agricultural production (thus creating surplus for sale);…The analysis of the impact on household income however reveals that there was no significant increase in total household income resulting from implementation of MVP. The overall income effect was insignificant, even though decomposing the income into various components yielded only a significant reduction in remittances.(26)…Thirdly, in Sauri the majority of households sell their produce immediately after harvest, when prices are low. Thus, they realize very low income gains. It is for this reason that the MVP introduced cereal banks, which were meant to cushion farmers against selling their produce at the time when prices are depressed. However, very few of the cereal banks that were created are currently operational and successful, due to lack of proper management and consequently lack of trust among cereal bank members.”

26 Bill November 29, 2011 at 8:56 pm

Cliff, I can also see your reading as well. I wish it were clearer.

27 question the question November 29, 2011 at 3:43 pm

“And generally, low trust leads to poverty.”

Wow, now there’s a leap of faith statement if ever I’ve read one.

28 Jane Shevtsov November 28, 2011 at 11:14 pm

Table 3 in the paper gives the data. For Millennium Villages, the mean total income is 4748.22 and the standard deviation is 3915.7. That’s pretty bad, but for control villages, the mean income is 4563.22 and the standard deviation is 11533.87! That kind of spread means the analysis has very low power. (It also means that mean and standard deviation isn’t the way to summarize this data set.) Specifically, according to the calculator at http://www.dssresearch.com/KnowledgeCenter/toolkitcalculators/statisticalpowercalculators.aspx gives a power of 10.7% for alpha=0.1 and 5.5% for alpha=0.05. And you cannot draw a negative conclusion from a study with low power!

29 Cliff November 29, 2011 at 9:29 am

Why does the high standard deviation mean the analysis has low power? That’s not a measure of error, right? Just variance in income?

So it seems like maybe it reduces variance in income at least?

30 DK November 29, 2011 at 9:49 am

when variance is high, the estimate of mean is less certain.

31 Cliff November 29, 2011 at 10:32 am

Ah, I see it is only 175 randomly selected households. Why not do more?

32 Jane Shevtsov November 29, 2011 at 1:54 pm

There’s another very important issue here — the data can’t be described by its mean and variance at all! A standard deviation that’s much larger than the mean indicates highly skewed data. (Both data sets are probably skewed, but the problem is particularly acute in the control group.) But t-tests are for normal distributions! It’s obvious that this data isn’t even in the same county as normal, so t-tests are out. The thing to do is to describe the datasets using medians, not means, and do a bootstrap test. Surely an economist should know about skewed distributions!

33 ad*m November 29, 2011 at 11:21 am

No. The study is adequately powered to detect a difference of 1200KShs. The difference between treatment and control group is less than that.

In medicine the next step would be a cost effectiveness study, and I bet you it would show that the money invested in the MVP per household is substantially larger than 1200KShs.

Aid cannot ever compensate for low trust.

34 Brett Keller November 29, 2011 at 11:49 am

Jane and ad*m — you might enjoy Michael Clemens comments on power / sample size in response to my question on the same: http://blogs.cgdev.org/globaldevelopment/2011/11/a-kenyan-economist-offers-the-first-independent-and-rigorous-evaluation-of-the-millennium-villages-project.php#comment-59955

35 Jane Shevtsov November 29, 2011 at 2:04 pm

First of all, even for that difference, the power is only 37.5% with alpha=0.1. That’s better than before, but not enough to draw a negative conclusion from lack of significance. Second, and more importantly, we have the problem of highly skewed data. It is simply not meaningful to describe such data using means, which is why you often hear about median rather than mean incomes of regions. As I said above, shouldn’t an economist know this?

36 Rahul Shastri November 29, 2011 at 12:47 am

I am not sure, the insignificant effect on income means the project is a failure; the study repeatedly states that the gains from increased agricultural productivity may have been utilized for self consumption; hence, the project may not have improved incomes but may have improved nutrition; just better nutritional outcomes may, in the future lead too better health, which in turn shall lead to better productivity and so on.
Would an outcome which would have increased incomes and worsened nutritional status been better?
I guess we should be using a more comprehensive measure of wellness, not just income.

37 TravisA November 29, 2011 at 1:31 am

In some ways, the analysis doesn’t make much sense. I guess I should read the paper. Farm productivity increases 70%. So farm income goes up. But other family members leave higher paying non-farm jobs to become farmers again and as a result total family income doesn’t increase. So that means farming activity must have generated lower income than non-farm jobs. If so, then why did people farm? Perhaps farming is a lower skill job, so that if you couldn’t work off the farm, you farmed.

Maybe five years isn’t a long enough time to measure the impact. I find it hard to believe that if farming productivity increases 70% every 5 years that total family income will continue to stay the same.

38 Rahul November 29, 2011 at 2:32 am

What examples do we have of third world nations bootstraping themselves out of poverty by agricultural means? If Sach’s intervention is indeed causing a shift from non-farm to farm activities, I’d be worried.

I’m skeptical that the nutritional outcomes of growing your own food are superior to those from other strategies. Western romanticism of the bygone pastoral lifestyles often handicaps these poverty projects by an excessive focus on farming when other (more industrial) economic activities might have been a quicker way out of the poverty trap.

39 Jeremy November 29, 2011 at 4:17 am

“I’m skeptical that the nutritional outcomes of growing your own food are superior to those from other strategies.” I’m equally skeptical that other strategies are any good at all in the context of the poor rural households that the MVP (and this evaluation) concern themselves with. A diversified diet, bringing in more locally grown and nutritionally rich crops, is likely to offer far greater and more sustainable increases in nutritional status and health than supplementation or fortification.

40 Rahul November 29, 2011 at 6:16 am

Were they consuming “locally grown” food it’d still reflect in incomes, because diverse produce needs to be bought (barter?) . This is distinct from growing “your own food”. I was following the line of reasoning of some other commentators that condone income stagnation by allowing for self-consumption.

41 TravisA November 29, 2011 at 12:06 pm

What happened in the past for other countries is irrelevant economically for the present Kenya. The question is whether agriculture is their comparative advantage. Given the high prices of food currently, it very well may be. Of course, this is all theorizing and I have no idea whether that is true for their geographical position.

42 Alan November 29, 2011 at 2:20 am
43 economics9698 November 29, 2011 at 3:26 am

Are there still Keynesians economists? Really? Idiots justifying the elites ripping off the peasants. Y = C + I – G + NX.

Fools and the fools web site.

How many Keynesians will there be 10 years from now when the Ponzi scheme has collapsed?

None.

You people are sooooo 2008. History.

2018 none of you will believe this shi.

44 Daniel November 29, 2011 at 4:16 am

Sounds like an economic geography problem to me. It’s always hard to make incomes rise in the periphery through direct measures unless you’re cutting the transport costs to the core and allowing the benefits of agglomeration to spillover into a wider area.

45 Bill November 29, 2011 at 10:26 am

That’s also a good point.

After I read this paper, I was thinking about how the West (as it existed in 1820) would have been developed, and what the incomes would have been, had not the railroads taken the grain eastward. In markets without transportation or storage, what you would see would have been home farm consumption.

It is strange–very strange–to see on an econ website the failure of others to see that development improves when you have markets and the institutions to support them. So, had the storage facilities been in place, income would be higher, for example, and there would be less home consumption of food. Second, the paper and the comments seemed strange by not recognizing the cause of another factor which the authors maintained limited productivity growth in both the control and non-control group: small plot size. Now the question to ask would be: would the development of storage, enabling a grain market, make not only the small farmer wealthier, but also more productive because he could acquire other small farms with the added money and thereby improve his productivity.

Markets matter. Institutions matter. But, apparently libertarians don’t see that.

46 Bill November 29, 2011 at 4:36 pm

the other thing I would like to know is what happens with the agricultural surplus that was created. The report says it was consumed by the household–but in tribal societies it could just as well be consumed in the home and distributed to other tribal members. I assume these people know what they are doing when they say “consumed in the household”, but if “consumed in the household” is the negative default option for “not sold on the market” I would be leery.

47 Cliff November 29, 2011 at 6:16 pm

Nice conclusion.

48 NK November 30, 2011 at 1:26 am

That’s exactly the argument that was ridiculed before (above where people said money wasn’t enough)!

According to you if only the government has taken care of storage facilities…

And then next time the mills, the bakeries, etc.

Funny logic, that.

49 Bill November 30, 2011 at 9:31 am

NK, I did some more research on storage facilities. When you say “government taking care of storage facilities” what you are assuming is that these are government facilities, when they are not. What they are are basically farmer coop storage facilities–owned by farmers or their tribes–which not only store grain, but also sell fertilizer and seed. This has been slow getting off the ground. It represents something new.

50 Justin November 29, 2011 at 12:17 pm

I think I saw a link to the following book here not so long ago; seems relevant: http://www.amazon.com/Just-Give-Money-Poor-Development/dp/1565493338

51 Claudia Sahm November 29, 2011 at 3:20 pm

Here’s a post from a development economics blog:
http://www.aviewfromthecave.com/2011/10/empire-strikes-back-sachs-vs-world.html
about the debate over the MVP. This link also provides some back and forth between Sachs and his critics.

52 k November 29, 2011 at 6:09 pm

To be honest, although I am not in favour of such intervention, the reactions to them do generate important, meaningful data.

If the data gathered here is made public, it might make for more nuanced analysis.

Maybe instead of claiming that RCTs show us how to do something, they can show us what happens when you do something, leaving to future work to figure out the mechanism (but how do you foresee the data requirements?).

53 Veracitor November 29, 2011 at 7:28 pm

Obviously there was no AD problem. Those people were living at the Malthusian limit. Although household cash income did not rise, I wouldn’t be surprised to read that consumption of foodstuffs did increase, though not enough to “matter” because in the short run increased fertility would absorb the increase in nutrition. To get through the “demographic transition” people need permanently increased income from industrialization, not temporarily increased income from agricultural subsidies. (If you drew a curve it would probably be the same curve as you would see after a famine, when fewer people enjoy higher agricultural incomes for a while, until they breed back up to the Malthusian limit again.)

54 Bill November 29, 2011 at 8:00 pm

There were agricultural subsidies in year 1, cut to 50% in year 2, and eliminated in year 3.

55 Bill November 29, 2011 at 8:17 pm

Interesting facts about storage and access to markets re the point about income being flat from selling in surplus seasons and not holding for later sales Here is some additional data from a Millenium V P re villages; note Sauri in terms of its difference in selling to local millers:

“In terms of profitability, the prices of maize and other crops also
increased to partially offset the rise in fertilizer prices. For example, the
price of maize in Sauri at harvest time rose from $120 t
1 in Year 1 to
$154 t
1 in Year 2 and to $196 t
1 in Year 3. But in Year 3, the overall
value of the harvested maize decreased compared to the Year 2 value
because of lower yields in Year 3; this drop occurred even though the
price of maize increased (Tables 4 and 6).
Peak prices for the crop usually occurred 6–9 months after harvest, at the
time when farmers are preparing for the following season. As expected, selling
the product severalmonths after harvest generated higher values than selling at
harvest time. The magnitudes of the difference between low and peak prices
varied between sites and seasons. For example, the value of the maize at peak
pricewas at least double the value of harvest in Pampaida;whereas, only a slight
increase of up to 45% was observed for maize in Bonsaaso and millet in Tiby.
There was a general increase in the value of both the harvest and peak prices in
2008 compared to other seasons as a result of the general global hike in prices of
food commodities observed during that year.
A major challenge in selling surplus crop has been to find buyers for crops
grown in remote areas, where markets and buyers are generally not
organized. For example, in Koraro and Bonsaaso which are located far from
large towns (Makelle and Kumasi, respectively), farmers were organized to
sell their agriculture products in bulk and the project facilitated the acquisition
of trucks to transport products to the markets; communities were trained to
manage the trucks and after an initial period of subsidy, the trucks were to be
maintained and run by the community. Various types of buyers were identified
for the sites: Tiby sells all its cereals at the annual cereal fair in Segou, the
nearby regional capital; Sauri and other clusters sell to local millers, and some
are selling to the World Food Program’s Purchase for Progress project, where
local foods are purchased for food aid in acutely hungry areas ( Jurry, 2008). http://mdg.ei.columbia.edu/east/sitefiles/file/Agronomy%20-%20Oct10.pdf

How much does this study turn on Sauri, its local millers, storage availability and functionality of storage.

Interesting.

56 Bill November 29, 2011 at 8:52 pm

Notice that if you deem additional surplus agriculture consumed at home, rather than sold, as income, there are income effects from MVP. It’s just that they didn’t treat the additional food consumed at home as income.

I always told my wife, when she stayed at home to raise the kids, that she was earning imputed income.

But, not for any Kenyan household survey.

57 Alex November 30, 2011 at 10:00 am

As far as I can tell this article does not seem to be published in a peer-reviewed journal. Peer-review offers oversight from experts that sign off that standard methodology is adhered to and findings don’t misrepresent data.

58 Nicole Radunsky November 30, 2011 at 10:15 am

I will leave my views on the Millennium Village Project aside here and speak only to a careful reading of this critique, which is deeply flawed. My major critiques of this paper are:

1. The researcher states that “The overall income effect was insignificant, even though decomposing the income into various components yielded only a significant reduction in remittances.” If the family is able to maintain the same income without remittances, and is freed from charity of family, this is a positive effect. (p. 26)

2. The author pretends in the abstract that the lack of effect is due to villagers moving their productive capacity to agriculture, but also states that “The results showed that MVP had an insignificant effect of agricultural dependency.” (p. 29)

3. Finally, the researcher states that the unexpected lack of income gain due to increased agricultural production is created by “a very high proportion of production allocated to self-consumption, including the production increment due to (even substantial) productivity gains.” (p. 32) However, in development economics we define income not by cash changing hands (since this means very little in subsistence, largely informal economies) but rather through surveys of household consumption and savings. An increase in consumption should be reflected in income just as much as an increase in sales of agricultural production.

This leaves aside the methodological limitations of the study. Of course, this paper is not peer reviewed, and in my opinion could never be published by a legitimate journal. A devastating blow may come to the Millennium Villages yet, but the response of those critical of Sachs should not be to jump on any poorly crafted and embarrassing bandwagon that pops up as a working paper.

Comments on this entry are closed.

Previous post:

Next post: