Michael Clemens won’t let up on the Millennium Villages + bonus links

It’s nice to see scholars bringing attention to the critical need for evaluation and informed public dialogue (not just “success stories” or short-term impact evaluation) for the Millennium Villages Project, which we have also covered on this blog. Michael Clemens of the Center for Global Development is currently carrying on a very revealing dialogue with Millennium Villages. In Michael’s first blog post which we blogged, he makes three central points:

1. The hundreds of thousands of people living in the Millennium Villages, present and future, deserve to know whether the project’s combination of interventions is backed up by good science.
2. Randomized evaluation is the best way to do this. While it may be too late to properly evaluate the first wave of villages, there is still time to conduct such a study for the next wave of villages.
3. The MVP evaluation should demonstrate long-term impact before it is scaled up.

In a subsequent post, Michael parses the curious non-answer he receives from the director of monitoring and evaluation for the MVP, Dr. Paul Pronyk. He breaks down—for those of us not intimately involved in the finer details of impact evaluation—the difference between true scientific evaluation and what the MVP says it is doing, namely “matched randomly selected comparison villages.”

What the MVP has done is something very different from…a rigorous evaluation.  First, village cluster A1 was chosen for treatment, for a range of reasons that may include its potential for responding positively to the project.  Then, long after treatment began, three other clusters that appear similar to A1 were identified — call these “candidate” comparison clusters A2, A3, and A4.  The fact that all three candidates were chosen after treatment in A1 began creates an enormous incentive to pick those candidates, consciously or unconsciously, whose performance will make the intervention in A1 look good.  Then the comparison village was chosen at random from among A2, A3, and A4.

Differences between the treated cluster and the comparison cluster might be due to the MVP. But those differences might also be due to how the original Millennium Village was chosen, and how the three candidate comparison villages were chosen.  This is not a hypothetical concern…

So, either the MVP director of evaluation does not understand evaluation...or he thinks we won't know the difference.

Dr. Pronyk promises the release of the MVP’s midpoint evaluation at some unspecified time later this year, and said they “look forward to an active discussion about the initial findings regarding poverty, hunger, and disease in the Millennium Villages.” We hope the scholarly community and the wider reading public concerned with development issues will give Dr. Pronyk precisely what he’s asking for.

Bonus Links

* Sounds a bit like a parody we wish we’d written….but it’s true. Yesterday’s NYT features this quote from a story on China’s bid to supply California with technology, equipment and engineers to build a high-speed railway, and to help finance its construction:

“We are the most advanced in many fields, and we are willing to share with the United States,” Zheng Jian, the chief planner and director of high-speed rail at China’s railway ministry, said.

* We’d be remiss not to mention this helpful timeline of celebrity aid to Africa featuring an interactive map from Mother Jones (and some additional commentary from Wronging Rights and Texas in Africa.)

The best way nobody’s talking about to help Haitians

The following post is by Michael Clemens, a research fellow at the Center for Global Development in Washington, DC, and an affiliated associate professor of public policy at Georgetown University.

The earthquake two weeks ago hit Haiti hard because Haiti is poor. The rich U.S. had similar earthquakes with far less carnage. So, what would do the most to lift Haitians out of poverty?

Start here: What has done the most, to date, to lift Haitians out of poverty? That answer is easy. Leaving Haiti brought more Haitians out of poverty than anything else that has ever been tried: any aid project in Haiti, or any trade preference for Haiti. See my note and video posted the day before Haiti’s catastrophe.

Of all the Haitians who live either in the United States or Haiti, and who live on more than $10 per day—at U.S. prices, adjusted for the fact that things are cheaper in Haiti—how many live in the U.S.? (That’s a barebones poverty standard, just one third of the U.S. “poverty line” for a single adult.)

82 Percent of Haitians above this poverty line are here in the United States. (I calculate this with Lant Pritchett here, ungated version here.) Only the top 1.4 percent of people in Haiti had that living standard even before the quake, and there is no evidence that Haitian emigrants come primarily from the extreme tip-top of the income distribution. So for most of Haitians who left, leaving Haiti was the cause of leaving poverty.

The Obama administration decided that for the next 18 months it will not deport any Haitian. But the U.S. has only been deporting about 1,000 Haitians per year recently. More importantly, the U.S. has forcibly stopped and repatriated about 5,000 Haitians per year for the past 20 years—people who never made it to the U.S. And this policy surely deterred thousands more each year from even trying. When Gallup asked people in Haiti last year if they would leave permanently if given the opportunity, 52 percent said yes. The U.S. is actively blocking the most effective poverty reduction strategy for Haitians.

When I talk about leaving Haiti as a development strategy for Haitians, some thoughtful people argue that this “can’t be the solution for Haiti.” Compared to what we all wish for in Haiti—rapid emergence from poverty for everyone there, in their homeland—leaving Haiti is a terrible solution. But compared to what is actually likely to happen in Haiti, continued poverty for decades at least, leaving Haiti is the principal solution to poverty. This is the right comparison, not the comparison to a prosperous Haiti that must remain a fantasy for now.

The best thing the United States could do for Haitians would be to let them in, either temporarily or permanently. We are now accepting about 21,000 permanent Haitian immigrants per year, and just a few hundred temporary workers per year. If we really wanted to raise Haitians out of destitution, we could absorb many times more than this. To say that we shouldn’t because it wouldn’t be the end-all solution is like saying that a lifeboat shouldn’t fill its ten empty seats just because there are 100 people in the water.

Do Millennium Villages work? We may never know

Jeffrey Sachs’ Millennium Villages Project has to date unleashed an array of life-saving interventions in health, education, agriculture, and infrastructure in 80 villages throughout ten African countries. The goal of this project is nothing less than to “show what success looks like.” With a five-year budget of $120 million, the MVP is billed as a development experiment on a grand scale, a giant pilot project that could revolutionize the way development aid is done.

But are they a success? To address that question, we need to know: What kind of data is being collected? What kinds of questions are being asked? Three years into the start of one of the highest-profile development experiments ever, who’s watching the MVPs?

The most comprehensive evaluation of the project published so far is a review by the Overseas Development Institute, a large UK-based think tank. The review covered two out of four sectors, in four out of ten countries, with data collected in the MVs only, not in control villages. The report’s authors cautioned that “the review team was not tasked and not well placed to assess rigorously the effectiveness and efficiency of individual interventions as it was premature and beyond the means of the review.”

Despite this, a Millennium Villages blog entry on Mali says, “With existing villages showing ‘remarkable results,’ several countries have developed bold plans to scale up the successful interventions to the national level.” Millennium Promise CEO John McArthur described Sachs’ recent testimony to the Senate Foreign Relations Committee: “Sachs noted the success of the Millennium Villages throughout Africa and the tremendous development gains seen in the project over the past three years.”

The Evaluation that Isn’t?

In contrast, evaluation experts have expressed disappointment in the results they’ve seen from the Millennium Villages Project to date. This isn’t because the MVPs fail to produce impressive outcomes, like a 350 percent increase in maize production in one year (in Mwandama, Malawi), or a 51 percent reduction in malaria cases (in Koraro, Ethiopia). Rather, it has to do with what is—and is not—being measured.

“Given that they’re getting aid on the order of 100 percent of village-level income per capita,” said the Center for Global Development’s Michael Clemens in an email, “we should not be surprised to see a big effect on them right away. I am sure that any analysis would reveal short-term effects of various kinds, on various development indicators in the Millennium Village.” The more important test would be to see if those effects are still there—compared with non-Millennium Villages—a few years after the project is over.

Ted Miguel, head of the Center of Evaluation for Global Action at Berkeley, also said he would “hope to see a randomized impact evaluation, as the obvious, most scientifically rigorous approach, and one that is by now a standard part of the toolkit of most development economists. At a minimum I would have liked to see some sort of comparison group of nearby villages not directly affected by MVP but still subject to any relevant local economic/political ‘shocks,’ or use in a difference-in-differences analysis.” Miguel said: “It is particularly disappointing because such strong claims have been made in the press about the ’success’ of the MVP model even though they haven't generated the rigorous evidence needed to really assess if this is in fact the case.”

An MVP spokesperson told me that they are running a multi-stage household study building on detailed baseline data, the first results from which will be published in 2010. The sample size is 300 households from each of the 14 MV “clusters” of villages (which comprise about 30,000-60,000 people each.) She also said that their evaluation “uses a pair-matched community intervention trial design” and “comparison villages for 10 MV sites.”

But Jeff Sachs noted in a 2006 speech that they were not doing detailed surveying in non-MV sites because—he said— “it’s almost impossible—and ethically not possible—to do an intensive intervention of measurement without interventions of actual process.” A paper the following year went on to explain that not only is there no selection of control villages (randomized or otherwise), there is also no attempt to select interventions for each village randomly in order to isolate the effects of specific interventions, or of certain sequences or combinations of interventions.

CEO John McArthur declined to comment on this apparent contradiction. The MVP spokesperson could say only that the evaluation strategy has evolved, and promised a thorough review of their monitoring and evaluation practices in 2010.

Comparison villages could be selected retroactively, but the MVP has failed to satisfactorily explain how they chose the MVs, saying in documents and in response to our questions only that they were “impoverished hunger hotspots” chosen “in consultation with the national and local governments.” If there was no consistent method used in selecting the original villages (if politics played a role, or if villages were chosen because they were considered more likely to succeed), it would be difficult to choose meaningful comparison villages.

Living in a Resource-Limited World

Imagine that you are a policymaker in a developing country, with limited resources at your disposal. What can you learn from the Millennium Villages? So far, not very much. Evaluations from the MVP give us a picture of how life has changed for the people living in the Millennium Villages, and information about how to best manage and implement the MVP.

Sandra Sequeira, an evaluation expert at London School of Economics, sums up the quandary neatly. “Their premise is that more is always better, i.e. more schools, more clinics, more immunizations, more bed nets. But we don't live in a world of unlimited resources. So the questions we really need to answer are: How much more? Given that we have to make choices, more of what?”

These are tough questions that the Millennium Villages Project will leave unanswered. For a huge pilot project with so much money and support behind it, and one that specifically aims to be exemplary (to “show what success looks like”), this is a disappointment, and a wasted opportunity.

How Nice Should Aid Commentators Be?

I wanted to respond today to your very helpful comments on yesterday’s launch, but of course I have to be very selective. To summarize a few areas of agreement and disagreement: I agree with:

(1) Those who said they liked the new blog. You get a free cup of coffee made with my hand-powered $20 espresso maker next time you are in Greenwich Village.

(2) Lucas who said I do need positive examples of aid working. Yes! Please send me more documentation on the Filipino example you gave, and I am happy to feature it. Positive examples are welcome from everyone reading this (but some kind of evidence and documentation is required.)

(3) Michael Clemens of the Center for Global Development on the counterproductive fixation with “0.7 percent of GDP” as an aid target. He is too modest – what he says is based on a killer article he did with Todd Moss also of CGD. The journal summary practically burns up the page:

First, the target was calculated using a model which, applied to today's data, yields ludicrous results. Second, no government ever agreed in a UN forum to actually reach 0.7 per cent – though many pledged to move toward it….The 0.7 per cent goal has no modern academic basis, has failed as a lobbying tool, and should be abandoned.

Clemens and Moss might have been a good reference to check before two opeds by Mr. Zoellick that mentioned “0.7” five separate times.

I disagree with:

(1) Jim, who said I was being too mean to Mr. Zoellick. First, I won’t be mean to YOU, Jim, I’m happy you gave me some tough criticism, debate is a GOOD thing.

Which is also my response to your criticism, which is that debate is a GOOD thing. Debate is good in academia, and it’s good in politics, and both kinds are usually fierce. It wasn’t a personal attack on Mr. Zoellick, it was a big disagreement about big issues.

We fiercely debate domestic spending bills that waste affluent taxpayers’ money with a few millions on a bridge to nowhere, so why should we be NICE when the head of the world’s premier aid agency outlines virtually zero accountability for helping the world’s poorest people?