Dear UK Government, why won't you let me retire as Official Sachs Critic?

UPDATE 3: FEB 8 4:50PM: Twitter War reveals that Millennium Village Blog accused Clemens and Demombynes of hard hearts towards suffering (search the blog for "suffering"). UPDATE 2: FEB 8 4:30PM: concluding coverage by @PSIHealthyLives @viewfromthecave of the Great Twitter War prompted by this post between @aidwatch and @earthinstitute, with collateral attacks on @m_clem, ending in a non-acceptance of debate by head of @earthinstitute.

UPDATE Feb 8 11:30am: sent comment with link to this post to DFID Independent Commission for Aid Impact. Spokesperson promptly responded, rejected my comment for consideration on technical grounds, but did warmly invite me to complete the anonymous online mass Survey Monkey. It probably doesn't mean much, unless the Independent Commission already learned the brilliant strategy of bureaucratizing the critics? I do feel a wee bit sorry for one of your Commisioners, the great John Githongo, who presumably did not risk his life so he could be reading anonymous results from Survey Monkey.

Nobody is more tired of the interminable Sachs-Easterly debate than one guy named Easterly...alas, I seem to be stuck in a kind of Critic Trap, in which the ideas criticized keep reappearing unchanged, requiring equally unchanged criticisms, keeping me in chronic peril of taking myself way too seriously.

So it was with great weariness I heard the news that the British aid agency DFID (otherwise probably the best bilateral aid agency) is close to financing a brand new Millennium Village in northern Ghana, near Bolgatanga. I had hoped for something better from the new UK government, which had seemed like an improvement over the Blair and Brown ("We know the answers, just double aid") team .

As it happened, I passed by the proposed MVP site last summer. The proposed villages are right on the main road in one of the most NGO-intensive places anywhere (see the sign below, in which NGOs apparently own the region).

The usual critique that selection bias of the Millennium Villages makes evaluation  impossible may be somewhat relevant given the political realities that (1) the current government chose the villages for the MVP, (2) the incumbents have frequently promised to do more for the North, (3) the MVP came along and may be a high visibility way to keep that promise, and (4) ergo, the government will likely do everything possible to make the project succeed, showing nothing about scalability for thousands of villages elsewhere. In short, this new MV may be about as informative as my feeding my own children is informative on whether child nutrition programs work.

And how good is the track record of the MVP taking evaluation seriously? Michael Clemens and Gabriel Demombynes posted the following on the World Bank Africa blog last Friday:

In a June 2010 report called Harvests of Development, the Project claimed that the impacts of the project included expanded cell phone ownership.  For example, the MVP claimed that increases in cell phone ownership at the Ghana site were caused by the project, in this extract from page 91 of the MVP report:

This claim has little basis, because cell phone ownership has been expanding at about the same rate all around the MVP site in areas untouched by the project. ....

But on Tuesday, months after multiple discussions we’ve had with MVP leaders on our research, a post on the MVP’s blog restated the claim that the increase in mobile phone ownership at the intervention sites was caused by the Project...

{The Clemens and Demombynes paper does the same cell phone analysis with the same results in the MV of Sauri, Kenya.}

They were responding to a blog post on the MVP web site on February 2, 2011 as follows:

Sauri looks back on five years of success

Infrastructure: ... The proportion of households owning a mobile phone has increased four-fold....

In short, independent observers made an irrefutable argument that a claim was invalid, the MVP heard the argument, seemed to accept it, and then repeated the previous claim unchanged.

Or in other words, if nobody is listening to any evaluations anyway, if I am bored and I am boring everyone else, why should I want to be Official Sachs Critic any longer?

Messrs. Clemens and Demombynes, you may want to check out a new job opening...

Why development history matters for the Millennium Villages Project

by Ed Carr, Associate Professor in the Department of Geography at the University of South Carolina A growing volume of critical writing on the Millennium Villages project (MVP) includes blog posts, journalistic pieces, scholarly works, and, recently, one partial social impact study. Nearly all point to project outcomes that could have been avoided had the project seriously engaged with the long history of field-based experiences in development.

Here, I will focus on just one example: Because the MVP did not critically evaluate the effect of its own assumptions about what works in development, a conflict between project goals and the needs of the villagers has emerged in at least one site.

The MVP is part of Millennium Promise, an effort to make progress towards the Millennium Development Goals (MDGs).  As a result, the MVP framed its interventions around the MDGs.  For example, in 2005 the MVP website described community participation in this MDG-centric manner:

An open dialogue [between MDG-trained teams and villagers] will cover topics such as local problems as related to the MDGs, constraints and opportunities for achieving the MDGs at their village level, initial discussions on possible solutions and approaches for achieving the MDGs, and general impressions/consensus on being included as a Millennium Villages Project site.

The project’s founders have stated that the MVP was built on the “core truth” that there are “known packages of effective and generally low-cost interventions” that can address poverty.  A review of the MVP described it as a pilot project seeking to “provide successful evidence of how to achieve the Millennium Development Goals”. The project’s focus on finding “successful evidence” for the efficacy of these packages of interventions suggests that the project has an interest in validating the importance of the problems identified in the MDGs and justifying the interventions of the MVP.

This creates a conflict of interest for the field staff of the MVP: What if the evidence does not show success? And what to do when the local community’s concerns do not align with either these solutions or the MDGs?

Those familiar with the history of development work know that such conflicts of interest are chronic. Take the classic by Robert Chambers:  Whose Reality Counts. He describes what happened when he examined a consultant's glowing report on a World Bank irrigation scheme and found evidence that the conclusions were wrong:

My points were more or less accepted, but then the matter was consigned to an indeterminate limbo.  Nothing was done.  Far from being rejected or modified, the consultant’s conclusions were published unchanged, and without reference to the criticisms....The consultants knew that the World Bank, which had commissioned the study, was keen to justify the new approach.  They knew what result was wanted.  Supported by the consultants’ unchanged report, the new approach was implemented on a large scale.  So, even if bad news is reported, it may be avoided, rejected or finessed out of sight. (p.82)

Another disconnect appeared in a UNDP/OECD evaluation of a project in Mali: “it has to be asked how the largely positive findings of the evaluations can be reconciled with the poor development outcomes (1985-1995) and the unfavorable views of local people.” (1999)

Similarly, a classic work by James Ferguson (1994) recounts a World Bank project to teach better farming techniques in a mountainous region of Lesotho, out of touch with local people who had long ago learned to abandon the poor soils of that region and work as migrants in South African mines.

There are the same significant pressures on the MVP field staff to press participants to conform to project assumptions and expectations, and to reject or finesse evidence and feedback that does not. Those designing and implementing the MVP should have addressed possible conflicts between their goals and those of the communities. They did not. As a result, I was not surprised to see this quote from a woman living in a Rwandan Millennium Village, from a recent study:

The MV has to meet with local community to learn more about what people really want because sometimes the MV brings things that the community doesn’t need or want.

This and several other issues with the MVP were easy to see from the outset (see here and here). But to recognize them required a familiarity with the history of development and a self-awareness that the Millennium Village Project itself has never shown.

--

Ed Carr is an associate professor in the Department of Geography at the University of South Carolina.  His book Delivering Development: Globalization’s Shoreline and the Road to a Sustainable Future was released by Palgrave Macmillan on February 1, 2011.  He blogs at Open the Echo Chamber.

Read all Aid Watch posts on the Millennium Villages project here.

What’s it like to live in a Millennium Village?

In Mayange, a cluster of villages about an hour’s drive south of Kigali, Rwanda, interventions by the Millennium Village Project across all sectors (in seeds, fertilizer, malaria nets, health clinics, vaccines, ambulances, water sources, classrooms, computers, cell towers, microloans and lots more) aim to lift villagers out of poverty within five to ten years. What do we know about the effects of such ambitious projects on the lives of the people living in these impoverished, rural communities? A qualitative study by Elisabeth King, a post-doctoral fellow with Columbia’s Earth Institute, produces a fascinating if limited* “snapshot” of social impacts of the Millennium Village Project in Mayange. A few observations:

The villagers King talked to were reluctant to bare all to a foreigner asking questions about delicate social topics. Her questions about quality of life, trust, and social exclusion elicited some contradictory and evasive answers: “Life in Mayange…In general it is not bad, it is not good, it’s in between.” “I know people well. But then, people are private and one only knows one’s own problems.” “There are no problems. But there are always some small problems between people though.”  King explained that in her previous research she found that Rwandans “downplayed negative aspects of social life and tended to embed negative reflections within positive pro-government ‘bookends.’”

MVP has good brand recognition and outreach; cooperatives sometimes increase cooperation. King found that the project was well-known among villagers, and almost all could name a change that had resulted from the project. Most were members of some kind of cooperative (farming, basket-weaving, bee-keeping) created by the project, and some described these groups as strengthening social bonds in the community or increasing women’s confidence by helping them provide income for their families.

Villagers thought that benefits from the project were unevenly distributed. In response to “Who gained the most from the project?” villagers answered most frequently “MV staff,” followed by “local leaders,” and villagers most willing to adopt new practices suggested by the project.

MVP may not be doing so well on the most basic thing – letting people say what THEY want. The most common suggestion was that the project should consult more with people in the community about what they want. One woman told King: “The MV has to meet with local community to learn more about what people really want because sometimes the MV brings things that the community doesn’t need or want. People may have good ideas.”

--

*King’s study is limited in several ways beyond lack of statistical significance (she spoke with 35 individuals and 8 focus groups in a population of 25,000 people). One, as a visiting Westerner asking questions about MVP, she can’t avoid being seen as tied with the project. Whether this makes her interviewees more timid in voicing complaints (for fear of losing some project benefit or subsidy), or more bold (in the hopes of gaining resources to address their troubles) is hard to say. Two, the Rwandan ban on talking about ethnic divisions prevents people from speaking candidly about this obvious issue in a place where resettled genocide survivors and released prisoners now live side by side. Three, King has no baseline data, so she can’t talk about changes in quality of life or social cohesion based on statements from before the project vs. during/after the project (see also: Clemens and Demombynes).

--

Thanks to Michael Clemens for the tweet that sent this study our way.

Read previous posts on the Millennium Villages here.

Eyes Wide Shut: Philanthropy Action on the "Rescheduled" Sachs vs. Clemens/Demombynes debate

Tim Ogden at Philanthropy Action issues a petition for the "rescheduled" (quotes in original) Sachs vs. Clemens/Demombynes debate on evaluating Millennium Villages, which was supposed to happen last Wednesday, to be indeed, well, rescheduled.

He asks for all of us to be watching whether this indeed happens. Aid Watch is always in favor of more Watching, so we support Tim's petition.

Millennium Villages: Moving the goalposts

Here on the blog, we’ve been following the progress of the Millennium Villages Project, a joint effort from the UN and Columbia’s Earth Institute that has introduced a package of development interventions in health, education, agriculture and infrastructure into 14 “clusters” of villages throughout 10 African countries. In response to a critical paper by Michael Clemens and Gabriel Demombynes, the MVP architects published a statement last week that they said would “clarify” some “basic misunderstandings” about the project. This statement caught our attention because—I would argue—what it is actually doing is seeking to reframe the debate about the project, and redefining project success in different, less ambitious terms.

“The primary aim” of the project, the MVP architects write, “is to achieve the Millennium Development Goals in the Project sites, as a contribution to the broader fulfillment of the MDGs (Evaluating the Millennium Villages: A response to Clemens and Demombynes, October 2010, emphasis in the original). Also important, they say, is to clarify what the MVP is not: “The MVP is not testing a rigid protocol for implementing MDG-based outcomes…The MVP is not claiming or aiming to provide a unique or “optimal” model for achieving the MDGs.”

This sounds fine unless you’ve read the many other MVP project reports and documents that clearly outline other, different, major goals and indicators of success.

For example:

• “To create and demonstrate a model for achieving the MDGs that can go to scale in all senses: concept, financing, implementation, monitoring and evaluation” (The Millennium Villages Scaling-Up Strategy,” late 2005)
• “To raise the capital stock above a threshold level, above which the village can move toward self-sustaining economic growth.” (The African Millennium Villages, PNAS, 2007)
• To prove that ”rural Africa can…escape the extreme poverty that traps hundreds of millions of people throughout the continent.” (Millennium Villages Info Kit, undated)
• To provide “a solution to extreme poverty” (Millennium Villages Info Kit, undated)

So, in this context, what’s even more revealing about this new statement is what it does NOT say. It does not mention that the improvements to the villages will be self-sustaining, or even moving towards self-sustainability by 2015, although that notion was at one point advertised as a “central proposition underpinning the Millennium Villages concept” (MVP FAQ, late 2006). In this case, the clarification seems more like a retrenchment, a moving away from the ambitious claims made at the project’s optimistic outset.

The new MVP definition also backs away from talking about interventions “undertaken as a single integrated project” that will serve as “proof of concept that the poverty trap can be overcome” (as stated in the PNAS paper cited above). In fact the impact of the project as an integrated whole can’t be demonstrated, the MVP authors argue, because some of the same improvements at work in the Millennium Villages (insecticide-treated bednets, subsidized fertilizer and seeds, for example) are also present in many of the surrounding villages.

Before, the project was defined in its own materials as a research experiment (a “proof of concept” carried out first in “research villages”) to prove that a package of development interventions delivered in a particular way can help lift the very poorest people living in rural Africa out of poverty forever. In today’s new formulation, the MVP is a means to show that by spending an amount roughly equal to 100 percent of the village’s per capita income on already “proven” interventions, for a period of 10 years, it can allow that village, for at least one moment in time in 2015, to step across the finish line demarcated by the Millennium Development Goals.

If the project continues to define success in these narrower terms, it will effectively shift the focus away from any obligation to show that the positive things achieved in the Millennium Villages are self-sustaining beyond the 10-year life of the project, or to prove that they are actually a result of the project itself.

POSTSCRIPT:

Screen shot of the top of World Bank's Africa Can...End Poverty Blog last Friday:

That notice was removed; here’s what the same blog tells us, at the bottom of the post, today:

UPDATE: Another view from Chris Blattman.

Millennium Villages: don't work, don't know or don't care?

UPDATE 10/16 12:25PM:  Tim Harford in FT also covers Clemens and Demombynes paper and gets response from Sachs. In a new paper, Michael Clemens and Gabriel Demombynes ask:

When is the rigorous impact evaluation of development projects a luxury, and when a necessity?

The authors study the case of the Millennium Villages, a large, high-profile, project originally meant to demonstrate that a package of technology-based interventions in education, health and agriculture could lastingly propel people living in the poorest African villages out of poverty within five (now ten) years.

One way Clemens and Demombynes get at their central question is to examine how the Millennium Villages are (so far) being evaluated, and ask whether a more rigorous method of evaluation would be 1) feasible and 2) likely to yield very different results. They answer 1) yes and 2) yes.

They start by looking at the findings of a Millennium Villages midpoint report released last summer, which shows movement on indicators (higher crop yield, more cell phone usage, fewer cases of malaria, etc.) against a baseline of data collected in those same villages three years prior. In the graph of cell phone ownership in Kenya below, this progress is charted by the black line.

Clemens and Demombynes then put this data in the context of how non-Millennium villages in the same country and region are faring on these same indicators, using publicly available data from national surveys. These are the red, blue, and green lines in the figure below.

What is going on in non-Millennium Villages in Kenya to drive up the number of cell phone users? Conventional wisdom is that it’s driven in very small part by outside aid and in large part by entrepreneurs, small and large. (There are a whole series of these graphs included in the paper, many show more improvement in MVs than in comparators, while a few show worse performance in MVs.)

The paper goes on to describe the weaknesses in the MVP’s published plans for future evaluations, which do involve comparison villages, and suggests how future waves of MVP interventions could be more rigorously evaluated without spending a lot more. (Summary here.)

The MVP team responded to the critique, saying that it “misunderstands the MVP’s aims and evaluation methods.” They shift away from portraying MVP as a demonstration project: the “primary aim” is “to achieve the Millennium Development Goals in the Project sites.”

Diary of a serial summit attendee

One week. Two development summits. Hundreds of heads of state, development luminaries, CEOs, and social entrepreneurs. Celebrity star power. No poor people. Aid Watch spent three days trying to make sense of the greatest show on earth to help the world’s lowest. TUESDAY

0930 hrs: I am crammed into a press box at the back of the world’s most glamorous development meeting, craning over the photographers to catch a glimpse of this year’s distinguished guests as they file into the room. At last, the charismatic master of ceremonies takes the stage, and the annual Bill Clinton Admiration Clinton Global Initiative comes to life. The meeting is to match people with big ideas with people with big money, and the pace of networking is furious.

1230 hrs: USAID administrator Raj Shah speaks at a CGI lunch on the topic of agriculture. While the Green Revolution saved hundreds of millions of lives in Asia, it never spread to Africa because aid agencies “actually just failed to try.” That doesn’t square with the World Bank’s finding that “Much energy has also been wasted in trying to replicate Asia’s Green Revolution model in Africa….”

1330 hrs: Introducing another “new” solution to world poverty, Hillary Clinton announced a $60 public-private partnership to replace dirty cooking stoves that spew toxic smoke with healthier, environmentally-responsible ones. (Read Alanna’s ideas on what this initiative will need to do differently to succeed where many previous efforts have failed, and these reflections from experience in India and rural Africa.)

1700 hrs: The best debate of the day is between Mohammed Yunus, who asks that the term microcredit not be used for firms that loan for profit, and Vikram Akula, of SKS Microfinance, who thinks only a commercial model can reach all the people who need and deserve loans, through access to capital markets. Here’s a summary from Forbes.

1830 hrs: My first “Tweetup,” at a bar in midtown, is much more fun than I anticipated. Lots of bloggers, aid workers, entrepreneurs and students whom I knew only by their Twitter handle now have faces and voices.

The best summary post of the day comes from Laura Seay, aka Texas in Africa, who articulates the uncomfortable sense that something essential is missing from these meetings:

… the presence of the poor is limited to pictures in slide shows while wealthy people hobnob over cocktails and abundant buffets. Am I the only one who would rather hear about what life as a poor woman in Ethiopia is like from an actual poor Ethiopian woman?

WEDNESDAY

0900 hrs: The “UN Digital Media Lounge” is where they keep the bloggers who couldn’t get real press credentials to attend the UN summit. There’s wifi, coffee and bagels, but at 47 blocks north of the actual UN building it feels a bit removed from the MDG summit. All day, different heads of state are speaking at the UN on “integrated and coordinated implementation of and follow-up to the outcomes of the major United Nations conferences and summits in the economic, social and related fields; and follow-up to the outcome of the Millennium Summit: draft resolution.” Some of these are broadcast on the screen at the lounge; I browse though others on the UN live feed site.

1430 hrs: The most hyped event of the day is the launch of a new global health strategy for child and maternal health, “Every Woman, Every Child.” Secretary General Ban Ki-Moon promised a “clear road map for making a fundamental difference in millions of lives.” Then he opened the floor to two-minute speeches from practically everybody in the room: poor countries, rich countries, foundations, corporations, NGOs, all making promises and pledges, which the UN announced amounted to “over $40 billion in resources for women and children’s health.”

Oxfam UK questions whether the funds pledged for women and children are actually additional funds or just “promises with a seemingly big price tag in a new shiny UN wrapper.”

1700 hrs: Meeting fatigue is setting in. Since I’m not invited to the MDG Gala, where attendees will celebrate pledges to fight poverty in New York’s swankiest Plaza Hotel, I’m grabbing a beer, going home, and watching President Obama’s speech from the comfort of my couch.

…Wait a minute, did President Obama really just admit the US approach to food aid is creating dependence, not development, and that our aid policies have focused on short term gains at the expense of sustainable development? Did he just become the world’s latest aid skeptic? Did he just pledge to be guided by evidence, “to invest in programs that work, and end those that don’t”? Judging from immediate reactions, people watching are starting to get that some of that old “Yes, we can” feeling.

THURSDAY

1045 hrs: It’s Raj Shah again, stopping by the UN Digital Media Lounge. Wow, did you know he’s only 37 years old, a medical doctor with a degree in health economics? The guy is impressive. But he doesn’t address the most obvious follow-up question to Obama’s speech last night: What happens next so that Obama’s hopey-changey speech gets translated into actual change in our 50-year-old aid legislation and at USAID and the 25 other government agencies involved in US foreign assistance? Will development really be elevated on par with diplomacy and defense when the White House’s new policy says that Shah will report to the Secretary of State, and will have a seat on the National Security Council only “as appropriate”?

1400 hrs: And, we’re back at CGI for a special panel on Haiti’s reconstruction. Uh-oh, is Haiti’s President René Préval really inviting Wyclef Jean on stage? President Clinton talks investment climate with the CEO of Royal Caribbean, the cruise line that brought in more than half of Haiti’s tourists last year. He describes a Coca-Cola/IDB/TechnoServe project sourcing Haitian mangos for a new Odwalla mango-lime juice, and speaks movingly about the resilience of the Haitian people.

Coca-Cola is everywhere this week, in the speeches of Melinda Gates, Raj Shah, in multiple panels at CGI. The prominence of corporations in this week’s events led to at least one wry comment about “saving the world with high-fructose corn syrup” and an observation that we’re hearing “more and more about mutual benefit and less about the moral requirement to help those in need.”

Given the overlap in timing, topics and headline speakers (Hillary Clinton, Mohammed Yunus, Ellen Johnson-Sirleaf, Melinda Gates and Pres Obama all spoke at both events), comparisons between the two events are inevitable, with one journalist suggesting the CGI could be “the new UN.” It does have far better production values, better food, and better (though still spotty) press access. Come to think of it, Bill Clinton would make a bit more inspirational SecGen than the mild-mannered one we have now. But let’s not forget that CGI members fork out $20,000 per year for the privilege to attend what is still, despite the roster of impressive accomplishments, a club for very privileged people.

1730 hrs: The leaders of the MDG summit have issued their “outcome document,” whose long stretches free of content, by custom, were agreed upon before the delegates even arrived:

We underscore the continued relevance of the outcomes of all major United Nations conferences and summits in the economic, social and related fields and the commitments contained therein, including the Millennium Development Goals….We strongly reiterate our determination to ensure the timely and full implementation of these outcomes and commitments.

Clinton is creating more of a pulse closing his show, which I’m watching from the press pen since they couldn’t fit half the press people into the mobbed closing session. Looking out at the audience (Oh my God that’s Mick Jagger!), Clinton quips that while “politics is show business for ugly people,” work in the non-governmental sector is “show businesses for nerds.” For this week, at least, he’s right, it’s been quite a spectacle. Thank goodness there’s 12 months until the next one.

If an evaluation is released on the internet and no one comments, does it make a sound?

The release of the Millennium Villages Project mid-point evaluation has so far been met with no discernable public response. Strange, since the release is billed as the “first major scientific report on progress after three years of MVP activity.” Doubly strange, since the MVP is an ambitious project that reaches into nearly all areas of its 500,000 recipients’ lives, and proposes, in scaled-up version, to completely change the architecture and delivery of aid to Africa.

So why the silence? Two possible reasons come to mind. Perhaps:

1. The evaluation doesn’t contain much that is unexpected or useful, and/or
2. No one really cares about evaluation.

We knew that the report would give the mid-point results of a longitudinal study comparing data from 300 Millennium Village families collected when the project began and again three, and five years later. (Although this is no longer the midpoint of anything, as the project has since expanded from 5 to 10 years.)

The new data give a picture of encouraging results across all sectors compared to the baseline. In Mwandama, Malawi, for example, bednet use for children under five increased from 14 percent to 60 percent and malaria prevalence for all age groups fell from 19 percent to 15 percent. Maize yields increased dramatically from .8 tons per hectare to 4.5 tons per hectare.

Such short-term results are positive in the sense that they describe real, immediate changes in the lives of thousands of very poor people. But they are not surprising given what we know about the level of resources and intensive technical expertise invested in these villages: the project doubles the size of the local economy—it is roughly equivalent to a 100 percent increase of per capita income per year (see here for calculations from Michael Clemens).

Unfortunately the results are also not that useful: Three years is too short a period to know how to interpret this dramatic increase in maize yields, for example. Is this consistent with normal variation in crop yields? Was 2006 an unusually good or bad year for maize? We don’t know.

The results also don’t help us determine whether current and future resources should be shifted away from other existing or even yet-to-be invented approaches, towards the MVP template. Will those short-term gains last beyond the timeline of the project? Can the project become self-sustaining?

Again, we don’t know, in part because not enough time has passed. Consider this anecdote from a New York Times blog series by Jeff Marlow on the Millennium Village of  Koraro, Ethiopia:

In 2005, all fertilizer was given away, leading to a significant increase in food production. Fertilizer subsidies were then progressively rolled back; by last year, only 50% of the cost was covered. For the 2009 growing season, the project tried something new: farmers were given loans for fertilizer, but they are expected to pay back the full cost plus interest when the harvest comes.

For many Koraro farmers, this is a daunting challenge. “The project used to help us with fertilizer,” says Brhana Syum…“But now it’s very expensive, and there’s no way to pay for it all.” Many farmers facing similar constraints have chosen to scale back their farms, thereby requiring less fertilizer, rather than face enormous debts…

So this particular push towards sustainability has come up against some obstacles. It may yet succeed, or it may fail. We don’t know the end of the story.

Supporters of the project argue that the individual interventions have already been proven: for example, we know that using better seeds and adding fertilizer will increase crop yield. But what the MVP says it is proving with this evaluation is the “value and feasibility of integrated community-based investments”—that is, the whole package of interventions, as well as the management systems used to deliver them. And this is precisely what the MVP does not have the data to demonstrate.

This evaluation repeats the call to scale up the project within existing project countries and expand to new ones, as quickly as possible. But the MVP as a whole remains an untested and unproven intervention, while the lives of Millennium Villagers—their habits, beliefs, livelihoods, and sources of authority—are  inevitably being changed in profound ways. This evaluation does nothing to change the argument of my previous post that the MVP should live up to their promise to be a ‘proof of concept:’ to be seriously and independently evaluated, and proven to work—beyond immediate short-term effects—before it is scaled up.

If you were sick and someone offered you a drug that hadn’t been tested, would you take it? And even if you would, would you want hundreds of millions of people whose lives depended on it to forego other types of treatment and take that drug too?

Jeff Sachs’ intellectual empire gets new funding

There’s a new way to study development: a masters degree in the practice of development. The MacArthur Foundation announced ten universities to receive funding for the new degree program yesterday, bringing the funding from MacArthur for this project to $16 million. The first students matriculated at Columbia University in 2009, and by 2013 the foundation expects the programs to be producing 400 graduates a year from around the world.

The two-year degree is multidisciplinary—the health sciences, the natural sciences and engineering, the social sciences, and management—with a focus on application and fieldwork.

Since today’s problems—like climate change, poverty and sustainable development—are interconnected, students need to be prepared to think across disciplines, so the argument goes. If ending global hunger (Millennium Development Goal number one) requires technical knowledge of health and nutrition, agronomy, agricultural supply systems, as well as managing organizational change, then this degree proposes to equip graduates with basic knowledge on all those topics.

The idea for the new global program comes from the Earth Institute’s Jeffrey Sachs, an architect of the Millennium Development Goals, and John McArthur, the head of the NGO that supports the Millennium Village Project, who articulated their vision in a 2008 report on education for development professionals.

Here’s what this program assumes the world needs more of:

a new generation of development practitioners who can understand the “languages” and practices of many specialties, and who can work fluidly and flexibly across intellectual and professional disciplines and geographic regions.

This sounds pretty good. In fact, I’m a generalist myself, which is how I ended up in this job, where I write about a global health issue one day and an economics paper the next.

But what if what the world really needs more of something else? What if it needs more specialists, more people with deep knowledge about the regions they study and work in? What if it needs people who are well-versed enough in their own disciplines to be critical of half-baked development ideas cooked up by aid planners who know just enough about every topic to believe they have the answers?  What if the world needs more specialists to evaluate the quality of the work in each specialty?

Curriculum and course materials proposed by the central “Secretariat” for development practice are housed in Columbia’s Earth Institute. Will the new programs produce students with a standardized, narrowly-prescribed view of how to approach development problems? Or will the melding of disciplines encourage critical thinking and help straddle the theory-policy divide, making global cooperation run more smoothly and international aid more effective?

I hope it’s the latter. But here’s one discouraging clue: The draft 2009 syllabus for the development practice degree’s required “foundation course,” offered at Columbia and several other universities around the world through web conferencing, reads like a synopsis of the degree itself. And all the readings for the course’s introductory week, the week devoted to foreign aid and policy, and the week on the Millennium Villages Project are authored by either Jeff Sachs, John McArthur, the Millennium Villages Project scientists, or the UN.

Hat tip to Michael Clemens.

Michael Clemens won’t let up on the Millennium Villages + bonus links

It’s nice to see scholars bringing attention to the critical need for evaluation and informed public dialogue (not just “success stories” or short-term impact evaluation) for the Millennium Villages Project, which we have also covered on this blog. Michael Clemens of the Center for Global Development is currently carrying on a very revealing dialogue with Millennium Villages. In Michael’s first blog post which we blogged, he makes three central points:

1. The hundreds of thousands of people living in the Millennium Villages, present and future, deserve to know whether the project’s combination of interventions is backed up by good science.
2. Randomized evaluation is the best way to do this. While it may be too late to properly evaluate the first wave of villages, there is still time to conduct such a study for the next wave of villages.
3. The MVP evaluation should demonstrate long-term impact before it is scaled up.

In a subsequent post, Michael parses the curious non-answer he receives from the director of monitoring and evaluation for the MVP, Dr. Paul Pronyk. He breaks down—for those of us not intimately involved in the finer details of impact evaluation—the difference between true scientific evaluation and what the MVP says it is doing, namely “matched randomly selected comparison villages.”

What the MVP has done is something very different from…a rigorous evaluation.  First, village cluster A1 was chosen for treatment, for a range of reasons that may include its potential for responding positively to the project.  Then, long after treatment began, three other clusters that appear similar to A1 were identified — call these “candidate” comparison clusters A2, A3, and A4.  The fact that all three candidates were chosen after treatment in A1 began creates an enormous incentive to pick those candidates, consciously or unconsciously, whose performance will make the intervention in A1 look good.  Then the comparison village was chosen at random from among A2, A3, and A4.

Differences between the treated cluster and the comparison cluster might be due to the MVP. But those differences might also be due to how the original Millennium Village was chosen, and how the three candidate comparison villages were chosen.  This is not a hypothetical concern…

So, either the MVP director of evaluation does not understand evaluation...or he thinks we won't know the difference.

Dr. Pronyk promises the release of the MVP’s midpoint evaluation at some unspecified time later this year, and said they “look forward to an active discussion about the initial findings regarding poverty, hunger, and disease in the Millennium Villages.” We hope the scholarly community and the wider reading public concerned with development issues will give Dr. Pronyk precisely what he’s asking for.

Bonus Links

* Sounds a bit like a parody we wish we’d written….but it’s true. Yesterday’s NYT features this quote from a story on China’s bid to supply California with technology, equipment and engineers to build a high-speed railway, and to help finance its construction:

“We are the most advanced in many fields, and we are willing to share with the United States,” Zheng Jian, the chief planner and director of high-speed rail at China’s railway ministry, said.

* We’d be remiss not to mention this helpful timeline of celebrity aid to Africa featuring an interactive map from Mother Jones (and some additional commentary from Wronging Rights and Texas in Africa.)

Beware the fury of a patient man: Michael Clemens on Millennium Villages

Michael Clemens at the Center for Global Development is a very calm,  judicious, sensible guy. But even he has finally lost patience with the lack of any serious evaluation of the Millennium Villages: Why a Careful Evaluation of the Millennium Villages is Not Optional

UPDATE (3/20, 8:16am) Chris Blattman adds his take on this.

New York Times on Millennium Villages

Jeffrey Gettleman reports today from Sauri, Kenya on the debate on the Millennium Villages.

Who gets the Last Seat on the Plane? Why Aid Hates Economics

Not long ago, I was returning home from a trip when the airline bumped me from my flight due to overbooking. The airline rep was very sympathetic, but I didn’t want her sympathy, I wanted A Seat On the Plane. She had traded off my wishes against those of other passengers, and I lost. Economists are unpopular because we say there is always SOME resource that is overbooked in aid, and aid is Forced to Choose: who is going to get the Last Seat on the Plane?

Politicians and advocates try to argue their way out of the Scarcity and Tradeoffs, using one or another of these proven strategies:

(1)   There really is no scarcity

This is Sachs’ central argument for more money in aid –you should never be forced to choose who should live and who should die, so you should always ask for more aid money. This has been effective as advocacy, but still doesn’t make aid money an infinite resource – there is still a limit on how much rich people will give. And the scarce resource is not only money – it is also political capital, rich peoples’ attention, or effective and accountable aid workers in the field. So using AIDS as an example, sure you should do some of both treatment and prevention – but how much of each? In the end, they are still competing for limited Seats on the Plane.

(2)   Our project doesn’t use any scarce resources

This argument is usually made by omission. The Millennium Villages don’t advertise that they are dependent on one extremely scarce resource -- Western experts -- perhaps it would then become obvious that they are neither scalable nor sustainable. And of course there is a big tradeoff between the Millennium Villages and better projects you could do with this scarce Western expertise. A better project replaces the scarce foreign expertise very soon with more abundant local expertise and labor – such as training programs to transmit foreign technical skills to locals, who will in turn pass it on to other locals.

(3)   My cause actually is the same as your cause

Advocates of one cause often argue many other causes NEED their cause. If the necessity is absolute, then indeed the tradeoff disappears. If it is less than 100 percent absolute, there is still a tradeoff. Hey, Other Passenger who took my seat: don’t claim that You are so Important that it’s pointless for Me to get on a plane without You! Unless You are the Pilot.

In summary, there really is scarcity and aid really is forced to make intelligent choices. Be sure to give a seat to the pilot.

Millennium Villages Comments, We Respond

We received the following comment this morning from the Director of Communications at the Earth Institute, regarding the Aid Watch blog published yesterday, Do Millennium Villages Work? We May Never Know. My response is below.----- It’s unfortunate that the author of this post chose to publish such an uninformed blog on the Millennium Village Project’s monitoring and evaluation activities. She and William Easterly at Aid Watch were invited to meet with our scientists and discuss the science and research behind the Villages and the details of the MVP monitoring and evaluation process before publishing any commentaries. Instead the author hastily chose to publish without talking with MVP researchers. The inaccuracy of the blogpost is a reflection of the lack of rigor and objectivity with which the Aid Watch authors approach this subject time and again.

For readers interested in reading factually accurate information about the Millennium Villages project and its monitoring and evaluation strategy, please see: http://www.millenniumvillages.org/progress/monitoring_evaluation.htm

Erin Trowbridge Director of Communications The Earth Institute

----- Dear Erin,

I had hoped for a different kind of response, one that addressed the specific points made by the piece. Your only comment on content is to say the piece was "uninformed." It would be helpful if you would clarify exactly what you think the piece got wrong, and offer what you view as the correct information to replace it. I would be happy to post such a response on Aid Watch.

Your comment is in any case an inaccurate characterization of our interaction over the past two months. I sent seven separate emails to you and one to CEO John McArthur beginning in mid-August, asking for information on the overall MV evaluation strategy, and eventually asking specifically for an explanation of how the thinking of the team had evolved from 2006 (when Jeff Sachs said there were no controls) to 2009 (when we were informed that there are comparison villages for 10 MV sites). Your responses were represented fairly in the blog post that Aid Watch published yesterday. We expressed willingness to meet with the research scientists after you offered this; it is unfortunate that we were unable to find a mutually convenient time to meet before our publication deadline, which we had already postponed several times.

Thank you for sharing further details of the MVP evaluation process with the information that has now appeared on the link you provided. Interested readers can now independently judge for themselves the merits and demerits of the ongoing MVP evaluation.

Frequent readers here may tire of hearing it, but it is our belief that greater transparency and a greater willingness on the part of donors and aid practitioners to share information with supporters and skeptics alike will make aid better.

Laura

Laura Freschi Associate Director Development Research Institute

Can We Push for Higher Expectations in Evaluation? The Millennium Villages Project, continued

There's been some good discussion—here in the comments of yesterday’s post and on other blogs—on the Millennium Villages and what sort of evaluation standard they can (realistically) and should (ideally) be held to. Yesterday on Aid Thoughts, Matt was distressed that over 70 percent of the student body at Carleton University voted in a tuition hike—$6 per student, per year, every year—to fund the Millennium Villages. The students apparently concluded that the MV program "offers the most effective approach for providing people with the tools they need to lift themselves out of extreme poverty. It is also the only approach that, if scaled, would see countless lives saved, an African Green Revolution and, amongst other things, every child in primary school."

How is it that students are coming away with that glowing impression from a project that—as Matt points out—has yet provided little evidence that its benefits are scalable, sustainable, persistent or transferable?

Focusing in on results published on one specific MVP intervention, blogger Naman Shah pointed us to his analysis of the MVP scientific team’s early malaria results.  The project's claims to have reduced malaria prevalence were “disproportionate to the evidence (especially given the bias of self-evaluation)” and suffered from some “bad science” like a failure to discuss pre-intervention malaria trends.

Chris Blattman stepped into the wider debate to offer an evaluator's reality check, questioning whether rigorous evaluation of the MVP is feasible.  Chris said:

[T]here are other paths to learning what works and why. I’m willing to bet there is a lot of trial-and-error learning in the MVs that could be shared. If they’re writing up these findings, I haven’t seen them. I suspect they could do a much better job, and I suspect they agree. But we shouldn’t hold them to evaluation goals that, from the outset, are bound to fail.

But if the industry standard of best practice is moving towards funding interventions that are measurable and proven to work, why is the MVP encouraging the international community to shift limited aid resources towards a highly-visible project apparently designed so that it can't be rigorously evaluated?

Fact is, none of us know exactly what kind of evaluation the Millennium Villages Project is doing, or the reasoning behind why they’re doing what they’re doing, since they haven't yet shared it in any detail.  Perhaps someone at the MVP will respond to our request to weigh in.

Do Millennium Villages work? We may never know

Jeffrey Sachs’ Millennium Villages Project has to date unleashed an array of life-saving interventions in health, education, agriculture, and infrastructure in 80 villages throughout ten African countries. The goal of this project is nothing less than to “show what success looks like.” With a five-year budget of $120 million, the MVP is billed as a development experiment on a grand scale, a giant pilot project that could revolutionize the way development aid is done.

But are they a success? To address that question, we need to know: What kind of data is being collected? What kinds of questions are being asked? Three years into the start of one of the highest-profile development experiments ever, who’s watching the MVPs?

The most comprehensive evaluation of the project published so far is a review by the Overseas Development Institute, a large UK-based think tank. The review covered two out of four sectors, in four out of ten countries, with data collected in the MVs only, not in control villages. The report’s authors cautioned that “the review team was not tasked and not well placed to assess rigorously the effectiveness and efficiency of individual interventions as it was premature and beyond the means of the review.”

Despite this, a Millennium Villages blog entry on Mali says, “With existing villages showing ‘remarkable results,’ several countries have developed bold plans to scale up the successful interventions to the national level.” Millennium Promise CEO John McArthur described Sachs’ recent testimony to the Senate Foreign Relations Committee: “Sachs noted the success of the Millennium Villages throughout Africa and the tremendous development gains seen in the project over the past three years.”

The Evaluation that Isn’t?

In contrast, evaluation experts have expressed disappointment in the results they’ve seen from the Millennium Villages Project to date. This isn’t because the MVPs fail to produce impressive outcomes, like a 350 percent increase in maize production in one year (in Mwandama, Malawi), or a 51 percent reduction in malaria cases (in Koraro, Ethiopia). Rather, it has to do with what is—and is not—being measured.

“Given that they’re getting aid on the order of 100 percent of village-level income per capita,” said the Center for Global Development’s Michael Clemens in an email, “we should not be surprised to see a big effect on them right away. I am sure that any analysis would reveal short-term effects of various kinds, on various development indicators in the Millennium Village.” The more important test would be to see if those effects are still there—compared with non-Millennium Villages—a few years after the project is over.

Ted Miguel, head of the Center of Evaluation for Global Action at Berkeley, also said he would “hope to see a randomized impact evaluation, as the obvious, most scientifically rigorous approach, and one that is by now a standard part of the toolkit of most development economists. At a minimum I would have liked to see some sort of comparison group of nearby villages not directly affected by MVP but still subject to any relevant local economic/political ‘shocks,’ or use in a difference-in-differences analysis.” Miguel said: “It is particularly disappointing because such strong claims have been made in the press about the ’success’ of the MVP model even though they haven't generated the rigorous evidence needed to really assess if this is in fact the case.”

An MVP spokesperson told me that they are running a multi-stage household study building on detailed baseline data, the first results from which will be published in 2010. The sample size is 300 households from each of the 14 MV “clusters” of villages (which comprise about 30,000-60,000 people each.) She also said that their evaluation “uses a pair-matched community intervention trial design” and “comparison villages for 10 MV sites.”

But Jeff Sachs noted in a 2006 speech that they were not doing detailed surveying in non-MV sites because—he said— “it’s almost impossible—and ethically not possible—to do an intensive intervention of measurement without interventions of actual process.” A paper the following year went on to explain that not only is there no selection of control villages (randomized or otherwise), there is also no attempt to select interventions for each village randomly in order to isolate the effects of specific interventions, or of certain sequences or combinations of interventions.

CEO John McArthur declined to comment on this apparent contradiction. The MVP spokesperson could say only that the evaluation strategy has evolved, and promised a thorough review of their monitoring and evaluation practices in 2010.

Comparison villages could be selected retroactively, but the MVP has failed to satisfactorily explain how they chose the MVs, saying in documents and in response to our questions only that they were “impoverished hunger hotspots” chosen “in consultation with the national and local governments.” If there was no consistent method used in selecting the original villages (if politics played a role, or if villages were chosen because they were considered more likely to succeed), it would be difficult to choose meaningful comparison villages.

Living in a Resource-Limited World

Imagine that you are a policymaker in a developing country, with limited resources at your disposal. What can you learn from the Millennium Villages? So far, not very much. Evaluations from the MVP give us a picture of how life has changed for the people living in the Millennium Villages, and information about how to best manage and implement the MVP.

Sandra Sequeira, an evaluation expert at London School of Economics, sums up the quandary neatly. “Their premise is that more is always better, i.e. more schools, more clinics, more immunizations, more bed nets. But we don't live in a world of unlimited resources. So the questions we really need to answer are: How much more? Given that we have to make choices, more of what?”

These are tough questions that the Millennium Villages Project will leave unanswered. For a huge pilot project with so much money and support behind it, and one that specifically aims to be exemplary (to “show what success looks like”), this is a disappointment, and a wasted opportunity.

Five simple principles for scaling up in aid

There is a lot of discussion in aid on scaling up small-scale successes in aid to reach many more potential beneficiaries. But what things can be scaled up? Here are some principles so simple that they would be embarrassing except that they are routinely violated in aid. (1) Scale up success not failure

The only reason for mentioning this is that the aid business has a strange habit of trying to scale up again things that have already failed. PROGRESA is the great success story of scaling up something after you had determined it was successful.

(2) Don’t scale up what you think is most important, scale up what you do best

There are lots of important issues, so why not choose the one that you do best? And let the people who are good at the other issues work on the other issues? Yet both official agencies and NGOs are often pulled away from what they do best by well-meaning politicians and funders who are focused only on final goals.

(3) You can scale up only what requires cheap, abundant inputs; you cannot scale up something that depends on expensive, scarce inputs

This is one of my problems with the Millennium Villages – they at least partly depend on world-class experts flying in to solve idiosyncratic problems of each village. World-class experts are a scarce resource that you can’t scale up.

(4) Things that you make routine are among the easiest to scale up

It worked for Henry Ford, McDonald’s, and WalMart, why not in aid? One of the secrets to success of the large vaccination campaigns that reduced child mortality was that relatively unskilled medical workers (in abundant supply) could give vaccinations as a routine activity. Of course, not everything can be made routine. For a more complex discussion about social service delivery in general, see the great paper by Lant Pritchett and Michael Woolcock.

(5) Evaluate whether you are still successful after scaling up

Scaling up often changes the nature of what you are doing, so evaluate whether the scaled-up version works as well as the original version.

I'm sure readers have other principles to suggest -- please do so!

Should starving people be tourist attractions?

Senegalese entrepreneur Magatte Wade on the Huffington Post touched a raw nerve about condescension towards Africans. She noted that a tourism operator was marketing one of Jeff Sachs’ Millennium Villages (MVs) as a vacation destination and quoted from the brochure "Please do not give anything to the villagers -- no sweets.”

I decided to look more into the MV tourism project, not to pile on, but because I believe patronizing attitudes towards Africans is a BIG issue in aid. The web site gives this introduction:

The Millennium Village Tour is a unique experience that introduces the … poverty traps in south-eastern Rwanda and the successful intervention package of the UN Millennium Villages Project.

I agree with Wade that it is dehumanizing that the villagers are just exhibits for tourists teaching them about abstractions like “poverty traps,” and are also to be used as propaganda for the MVs’ “successful intervention.”

The brochure that bothered Wade really is cringe-inducing, including also this line:

Please do not eat or drink in public. Many people in Bugesera District are still suffering from malnutrition…

If the MV is so successful, why are people still starving? Instead of worrying about hiding their food, why don’t the tourists pitch in on some MV project that helps the starving get food and nutritional supplements?

The tourism company offering the Rwanda MV tour is called EOS Visions and is headed by some German professionals. They have country subsidiaries, and it was the Rwanda one (staffed by Rwandans) that offered the MV tour. There are some benefits for the villagers as the company advertises 70 percent profit sharing with the local community. Obviously, there were some good intentions here. It’s never easy to negotiate encounters between very rich and very poor people, and some might think that these quotes from a tourist project are a minor issue.

The real problem is that patronizing attitudes towards the African beneficiaries of the MVs follow naturally from the ideas that inspire the MVs – that the poor are helpless victims and it is up to foreigners with superior expertise and funds to rescue them. Condescension towards Africans is both offensive AND a sign of a counterproductive approach to development.

Try looking at the poor Rwandans living in the MV not as anonymous and interchangeable exhibits for a “poverty trap,” but as individuals who possess rights and human dignity just like us. Then we maybe we will understand that the most impressive, knowledgeable, and motivated soldiers in the war on poverty are usually poor individuals themselves.