If an evaluation is released on the internet and no one comments, does it make a sound?

The release of the Millennium Villages Project mid-point evaluation has so far been met with no discernable public response. Strange, since the release is billed as the “first major scientific report on progress after three years of MVP activity.” Doubly strange, since the MVP is an ambitious project that reaches into nearly all areas of its 500,000 recipients’ lives, and proposes, in scaled-up version, to completely change the architecture and delivery of aid to Africa.

So why the silence? Two possible reasons come to mind. Perhaps:

1. The evaluation doesn’t contain much that is unexpected or useful, and/or
2. No one really cares about evaluation.

We knew that the report would give the mid-point results of a longitudinal study comparing data from 300 Millennium Village families collected when the project began and again three, and five years later. (Although this is no longer the midpoint of anything, as the project has since expanded from 5 to 10 years.)

The new data give a picture of encouraging results across all sectors compared to the baseline. In Mwandama, Malawi, for example, bednet use for children under five increased from 14 percent to 60 percent and malaria prevalence for all age groups fell from 19 percent to 15 percent. Maize yields increased dramatically from .8 tons per hectare to 4.5 tons per hectare.

Such short-term results are positive in the sense that they describe real, immediate changes in the lives of thousands of very poor people. But they are not surprising given what we know about the level of resources and intensive technical expertise invested in these villages: the project doubles the size of the local economy—it is roughly equivalent to a 100 percent increase of per capita income per year (see here for calculations from Michael Clemens).

Unfortunately the results are also not that useful: Three years is too short a period to know how to interpret this dramatic increase in maize yields, for example. Is this consistent with normal variation in crop yields? Was 2006 an unusually good or bad year for maize? We don’t know.

The results also don’t help us determine whether current and future resources should be shifted away from other existing or even yet-to-be invented approaches, towards the MVP template. Will those short-term gains last beyond the timeline of the project? Can the project become self-sustaining?

Again, we don’t know, in part because not enough time has passed. Consider this anecdote from a New York Times blog series by Jeff Marlow on the Millennium Village of  Koraro, Ethiopia:

In 2005, all fertilizer was given away, leading to a significant increase in food production. Fertilizer subsidies were then progressively rolled back; by last year, only 50% of the cost was covered. For the 2009 growing season, the project tried something new: farmers were given loans for fertilizer, but they are expected to pay back the full cost plus interest when the harvest comes.

For many Koraro farmers, this is a daunting challenge. “The project used to help us with fertilizer,” says Brhana Syum…“But now it’s very expensive, and there’s no way to pay for it all.” Many farmers facing similar constraints have chosen to scale back their farms, thereby requiring less fertilizer, rather than face enormous debts…

So this particular push towards sustainability has come up against some obstacles. It may yet succeed, or it may fail. We don’t know the end of the story.

Supporters of the project argue that the individual interventions have already been proven: for example, we know that using better seeds and adding fertilizer will increase crop yield. But what the MVP says it is proving with this evaluation is the “value and feasibility of integrated community-based investments”—that is, the whole package of interventions, as well as the management systems used to deliver them. And this is precisely what the MVP does not have the data to demonstrate.

This evaluation repeats the call to scale up the project within existing project countries and expand to new ones, as quickly as possible. But the MVP as a whole remains an untested and unproven intervention, while the lives of Millennium Villagers—their habits, beliefs, livelihoods, and sources of authority—are  inevitably being changed in profound ways. This evaluation does nothing to change the argument of my previous post that the MVP should live up to their promise to be a ‘proof of concept:’ to be seriously and independently evaluated, and proven to work—beyond immediate short-term effects—before it is scaled up.

If you were sick and someone offered you a drug that hadn’t been tested, would you take it? And even if you would, would you want hundreds of millions of people whose lives depended on it to forego other types of treatment and take that drug too?