Imagine potential aid recipients saying what THEY think

Twenty minutes outside the small town of Masindi, Uganda is a village called Kikuube…The local council member representing the village is none other than my Mum…I was surprised that she—as a village leader—had never heard of the MDGs. Yet she goes about her day fulfilling tasks meant to improve the welfare of her community; from educating her community about the use of bed nets, to regular home inspections enforcing sanitation codes, to empowering women with micro-loan programs. What does it say about the MDGs when the very people that are supposed to be beneficiaries don’t even know about them?

This quote comes from TMS (Teddy) Ruge, co-founder of Project Diaspora, an organization that involves Africans abroad in sustainable development initiatives in their home communities. His musings on his village leader Mum not knowing about the MDGs were inspired by this year’s United Nations MDG Week, a series of meetings and events in New York much more conducive to talking about all the good the West is doing for the Rest than hearing from the Rest.

(It also made us think about how the international aid orgs are now struggling to credibly include the voices of aid recipients—witness this cringe-inducing IMF video and website about how the IMF consulted with “civil society representatives” during their annual meetings. The carefully chosen quotes from the carefully chosen “representatives” in the video praised the IMF for its openness to dialogue with the ill-defined “civil society,” perhaps right after the IMF did some “capacity-building” on them.)

Teddy envisioned instead an event featuring just the voices of the potential aid recipients, a platform for people from his home village and others like it to share their experiences on getting by on $1 a day, and their successes in their own communities. Teddy first considered getting a TED license to build on their well-known brand, but decided to create his own model, which he is calling Villages in Action:

On Saturday, November 27, the microphone will be mounted stage center in this little quaint village. We welcome the world to join us in a frank discussion on the state of poor. We’ll discuss the MDGs and what our role is in achieving them by 2015 (and what we were already doing)...I want to challenge the notion that the sustainability of our communities depends on intervention from the West…

His idea to start public discussions about the development goals throughout villages in Africa is an intriguing one. He is looking for technical and financial support for this new project; find him here.

Eyes Wide Shut: Philanthropy Action on the "Rescheduled" Sachs vs. Clemens/Demombynes debate

Tim Ogden at Philanthropy Action issues a petition for the "rescheduled" (quotes in original) Sachs vs. Clemens/Demombynes debate on evaluating Millennium Villages, which was supposed to happen last Wednesday, to be indeed, well, rescheduled.

He asks for all of us to be watching whether this indeed happens. Aid Watch is always in favor of more Watching, so we support Tim's petition.

Millennium Villages: Moving the goalposts

Here on the blog, we’ve been following the progress of the Millennium Villages Project, a joint effort from the UN and Columbia’s Earth Institute that has introduced a package of development interventions in health, education, agriculture and infrastructure into 14 “clusters” of villages throughout 10 African countries. In response to a critical paper by Michael Clemens and Gabriel Demombynes, the MVP architects published a statement last week that they said would “clarify” some “basic misunderstandings” about the project. This statement caught our attention because—I would argue—what it is actually doing is seeking to reframe the debate about the project, and redefining project success in different, less ambitious terms.

“The primary aim” of the project, the MVP architects write, “is to achieve the Millennium Development Goals in the Project sites, as a contribution to the broader fulfillment of the MDGs (Evaluating the Millennium Villages: A response to Clemens and Demombynes, October 2010, emphasis in the original). Also important, they say, is to clarify what the MVP is not: “The MVP is not testing a rigid protocol for implementing MDG-based outcomes…The MVP is not claiming or aiming to provide a unique or “optimal” model for achieving the MDGs.”

This sounds fine unless you’ve read the many other MVP project reports and documents that clearly outline other, different, major goals and indicators of success.

For example:

• “To create and demonstrate a model for achieving the MDGs that can go to scale in all senses: concept, financing, implementation, monitoring and evaluation” (The Millennium Villages Scaling-Up Strategy,” late 2005)
• “To raise the capital stock above a threshold level, above which the village can move toward self-sustaining economic growth.” (The African Millennium Villages, PNAS, 2007)
• To prove that ”rural Africa can…escape the extreme poverty that traps hundreds of millions of people throughout the continent.” (Millennium Villages Info Kit, undated)
• To provide “a solution to extreme poverty” (Millennium Villages Info Kit, undated)

So, in this context, what’s even more revealing about this new statement is what it does NOT say. It does not mention that the improvements to the villages will be self-sustaining, or even moving towards self-sustainability by 2015, although that notion was at one point advertised as a “central proposition underpinning the Millennium Villages concept” (MVP FAQ, late 2006). In this case, the clarification seems more like a retrenchment, a moving away from the ambitious claims made at the project’s optimistic outset.

The new MVP definition also backs away from talking about interventions “undertaken as a single integrated project” that will serve as “proof of concept that the poverty trap can be overcome” (as stated in the PNAS paper cited above). In fact the impact of the project as an integrated whole can’t be demonstrated, the MVP authors argue, because some of the same improvements at work in the Millennium Villages (insecticide-treated bednets, subsidized fertilizer and seeds, for example) are also present in many of the surrounding villages.

Before, the project was defined in its own materials as a research experiment (a “proof of concept” carried out first in “research villages”) to prove that a package of development interventions delivered in a particular way can help lift the very poorest people living in rural Africa out of poverty forever. In today’s new formulation, the MVP is a means to show that by spending an amount roughly equal to 100 percent of the village’s per capita income on already “proven” interventions, for a period of 10 years, it can allow that village, for at least one moment in time in 2015, to step across the finish line demarcated by the Millennium Development Goals.

If the project continues to define success in these narrower terms, it will effectively shift the focus away from any obligation to show that the positive things achieved in the Millennium Villages are self-sustaining beyond the 10-year life of the project, or to prove that they are actually a result of the project itself.

POSTSCRIPT:

Screen shot of the top of World Bank's Africa Can...End Poverty Blog last Friday:

That notice was removed; here’s what the same blog tells us, at the bottom of the post, today:

UPDATE: Another view from Chris Blattman.

Millennium Villages: don't work, don't know or don't care?

UPDATE 10/16 12:25PM:  Tim Harford in FT also covers Clemens and Demombynes paper and gets response from Sachs. In a new paper, Michael Clemens and Gabriel Demombynes ask:

When is the rigorous impact evaluation of development projects a luxury, and when a necessity?

The authors study the case of the Millennium Villages, a large, high-profile, project originally meant to demonstrate that a package of technology-based interventions in education, health and agriculture could lastingly propel people living in the poorest African villages out of poverty within five (now ten) years.

One way Clemens and Demombynes get at their central question is to examine how the Millennium Villages are (so far) being evaluated, and ask whether a more rigorous method of evaluation would be 1) feasible and 2) likely to yield very different results. They answer 1) yes and 2) yes.

They start by looking at the findings of a Millennium Villages midpoint report released last summer, which shows movement on indicators (higher crop yield, more cell phone usage, fewer cases of malaria, etc.) against a baseline of data collected in those same villages three years prior. In the graph of cell phone ownership in Kenya below, this progress is charted by the black line.

Clemens and Demombynes then put this data in the context of how non-Millennium villages in the same country and region are faring on these same indicators, using publicly available data from national surveys. These are the red, blue, and green lines in the figure below.

What is going on in non-Millennium Villages in Kenya to drive up the number of cell phone users? Conventional wisdom is that it’s driven in very small part by outside aid and in large part by entrepreneurs, small and large. (There are a whole series of these graphs included in the paper, many show more improvement in MVs than in comparators, while a few show worse performance in MVs.)

The paper goes on to describe the weaknesses in the MVP’s published plans for future evaluations, which do involve comparison villages, and suggests how future waves of MVP interventions could be more rigorously evaluated without spending a lot more. (Summary here.)

The MVP team responded to the critique, saying that it “misunderstands the MVP’s aims and evaluation methods.” They shift away from portraying MVP as a demonstration project: the “primary aim” is “to achieve the Millennium Development Goals in the Project sites.”

“Proofiness:” trashing back on FAO hunger numbers

Just before the big UN meetings here in New York around the Millennium Development Goals, the FAO released new world hunger numbers, and Aid Watch listed reasons to worry that these numbers were “made up.” A blog post from Oxfam GB’s Duncan Green called our post “lazy and supercilious,” with the amusing headline “Easterly trashed.”  The accusation that I am “lazy” struck a raw nerve, and so I have responded forcefully by asking Laura to do more work.

A closer look at the FAO’s documents, along with information provided by smart Aid Watch commenters as well as the FAO’s own senior economist David Dawe validates, rather than “trashes,” many of the concerns Aid Watch raised.

For one, the methodology for the FAO survey numbers does not actually directly measure malnutrition but tries to estimate it indirectly based on a model of human calorie requirements and food availability and distribution:

From the total calories available, total calories needed for a given population, and the distribution of calories, one can calculate the number of people who are below the minimum energy requirement, and this is the number of undernourished people.

A modeled number is NOT the same as directly measuring malnutrition (as the WDI anthropometric numbers cited in the previous post attempt to do). Is the model correct? How did they test it? A model has many assumptions and parameters, which are inevitably less than 100 percent reliable. All of these make the modeled numbers subject to a LOT of uncertainty. Has FAO made any attempt to quantify the uncertainty? Have they tried comparing their estimates to the anthropometric measures in WDI?

Second, according to the FAO’s downloadable data charts, this exercise was last carried out in 2005-2007. These survey numbers are available for every country in the database (176 in all). The data tables tell us that while there is no country-level data for Iraq, Afghanistan, Somalia or Papua New Guinea, they are included in regional estimates, and there are country-level entries for places like Sudan, Zimbabwe and Libya for each three-year data collection period going back to 1990.

Neither of the most recent FAO State of Food Insecurity reports, from 2009 or 2008, includes discussions of the methodology for the 2005-2007 surveys. And neither explains how the 2008 figures were obtained. The 2009 report’s tables list as sources UN population data from 2006, and “FAO estimates” for undernourishment.

Third, the estimates for 2009 and 2010 are not only based on very indirect and noisy links between capital flows, imports, terms of trade and food availability, but the numbers for the former are not real numbers but based on USDA projected scenarios using IMF estimates for quantities that are notoriously difficult to estimate or project.

The comments from FAO economist David Dawe suggest (quite logically) that the economists, statisticians and policy-makers responsible for the FAO numbers are well aware of the drawbacks of the methodology they’ve chosen to produce both the survey year data and the estimates for years with no surveys at all; entire conferences and volumes are devoted to debating how to measure food deprivation.

Of course, none of this ambiguity and caution makes it into the papers. The New York Times reported simply that the UN said Tuesday, September 14 that the “the number of hungry people fell to 925 million from the record high of 1.02 billion in 2009,” but that “the level remains higher than before the 2008 food crisis.”

An alternative narrative based on the above would be something like: “the UN attempted on Tuesday to provide some projections for 2010 of the number of hungry people in the world compared to previous projections for 2009, all of which are in turn based on a combination of remarkably shaky links to other projections of impossible-to-project factors like capital flows, unverified and uncertain models of hunger and food availability, an unexplained estimate for 2008, and a survey of uncertain coverage and usefulness last conducted in 2005-2007.”

A new best-selling book called Proofiness opens with a quote that we are “vulnerable to the belief that any alleged knowledge which can be expressed in figures is in fact as final and exact as the figures in which it is expressed,” then the rest of the book explains why this “proofiness” is really “mathematical deception.”

Aid Watch will continue its lazy and supercilious attacks on proofiness.

Lant Pritchett on what Obama got right about development

by Lant Pritchett, Kennedy School of Government, Harvard University Obama's speech at the MDG conference and the announced US Global Development Policy are the result of long preparation and internal discussions within the administration as part of the Presidential Study Directive, lead out of the NSC, announced a year ago, and the QDDR, prepared by State, both processes having been watched over by the Washington think tanks and advocacy groups.

While one could immediately focus on the "architecture" part of the speech and read the Beltway tea leaves of who is up, who is down, and what that means for this organization or that, it is worth at least first stepping back and asking where this first official US development policy came down on the big debates on development, where, I think it comes out a big winner on four big ideas.

First, the speech and policy put economic growth front and center as objectives of development and development policy.  It might seem obvious that economic growth that increases people's command over resources is the single most powerful force to improve nearly any indicator of well-being -- from poverty to food security to health to education -- but, surprisingly, that point can get lost.  The "development is about more than growth" backlash, which had important elements of truth, easily got carried away into "development isn't at all about growth" and it is good to see economic growth back front and center of development objectives.

Of course growth these day must carry some adjectives as baggage -- "sustainable" and "broad based" can never be too far away -- but both of those are perfectly legitimate qualifiers and a small price to pay for the primacy of growth.

Second, the speech came down hard, and right, on the debate between improving systemic capability and programmatic action.  This was of course not easy to do in the context of a speech on the MDGs, which lend themselves to a programmatic vision of development.  Functioning systems of education have multiple and complex objectives -- spreading a common socialization, improving learning of the basics, identifying and promoting excellence.  The goal of development is that a country can have an education system that, as a natural part of its operation as a system composed of many actors and pressures, sets and achieves goals, some of which are then mapped into particular programs.  The same is true of all other spheres of social and governmental action -- infrastructure, law and order, health, economic policy.  The speech clearly identified building this capability as a central (and difficult) part of development.

This resists a very powerful tendency to reduce development to a series of specific targets, each of which can be addressed by the implementation of sufficiently resourced programs (programs which can be cocooned or stove piped around systemic dysfunction), which can be crudely caricatured as the "show me the money" approach.

The MDGs are correctly interpreted as what will be accomplished when there has been development -- not vice versa.

Third, the speech gets right the need for innovation, with rigorous evaluation as an important component of an environment for innovation.  The endeavor of "development" as a conscious acceleration of the progress of nation-states is now at least 50 years old.  If it were easy and obvious then as a social movement it would have disappeared under the weight of its own success and be a historical curiosity, like abolitionists.

The paradox of the external organizations that attempt to support development is that people tell them "we'll give you your budget if you tell us for sure what you are doing will work."  This leads to a powerful culture of pretending that much more is known about the "theory of change" that leads to development that really is known.  The fact that the wealthiest and most powerful country in the world has just spent eight years devoting fantastically high level of resources to "develop" Afghanistan (with security as one element of that) with results that range from mixed to shambolic should make it obvious that we need much greater openness within the development community to an approach of structured experimentation -- on all fronts.

The same skepticism about "one size fits all" that made "Washington Consensus" two dirty words should be taken to the range of "expert" advice in sectors from education to health to public sector governance to "institution building." All of which is mostly just repeating the conventional wisdom and closing off, rather than opening up, space for novelty and innovation.

Fourth, one thing the speech gets right it does so by omission.  There is no dollar figure.  The message "lets do more" is always popular because it also means "business as usual" for what is already going on.  "Let's do better at what we are doing" is a tough internal sell, but one that is useful -- including, I believe, to people who are actually on the ground, doing the work.  Anyone who has actually worked inside the development industry knows how much better things, at least potentially, could be, but also how tough achieving that will be given the inertia of massive organizations.  So while tackling policy and "architecture" might seem arcane relative to the apparent obvious gain of spending more money.  More is better, but better is better too and more is even better after better is better.

As to whether the proposed changes can advance the correct development agenda of growth, expanded public capability, innovation with evaluation, and improved assistance...well, that's a topic for another day.

Diary of a serial summit attendee

One week. Two development summits. Hundreds of heads of state, development luminaries, CEOs, and social entrepreneurs. Celebrity star power. No poor people. Aid Watch spent three days trying to make sense of the greatest show on earth to help the world’s lowest. TUESDAY

0930 hrs: I am crammed into a press box at the back of the world’s most glamorous development meeting, craning over the photographers to catch a glimpse of this year’s distinguished guests as they file into the room. At last, the charismatic master of ceremonies takes the stage, and the annual Bill Clinton Admiration Clinton Global Initiative comes to life. The meeting is to match people with big ideas with people with big money, and the pace of networking is furious.

1230 hrs: USAID administrator Raj Shah speaks at a CGI lunch on the topic of agriculture. While the Green Revolution saved hundreds of millions of lives in Asia, it never spread to Africa because aid agencies “actually just failed to try.” That doesn’t square with the World Bank’s finding that “Much energy has also been wasted in trying to replicate Asia’s Green Revolution model in Africa….”

1330 hrs: Introducing another “new” solution to world poverty, Hillary Clinton announced a $60 public-private partnership to replace dirty cooking stoves that spew toxic smoke with healthier, environmentally-responsible ones. (Read Alanna’s ideas on what this initiative will need to do differently to succeed where many previous efforts have failed, and these reflections from experience in India and rural Africa.)

1700 hrs: The best debate of the day is between Mohammed Yunus, who asks that the term microcredit not be used for firms that loan for profit, and Vikram Akula, of SKS Microfinance, who thinks only a commercial model can reach all the people who need and deserve loans, through access to capital markets. Here’s a summary from Forbes.

1830 hrs: My first “Tweetup,” at a bar in midtown, is much more fun than I anticipated. Lots of bloggers, aid workers, entrepreneurs and students whom I knew only by their Twitter handle now have faces and voices.

The best summary post of the day comes from Laura Seay, aka Texas in Africa, who articulates the uncomfortable sense that something essential is missing from these meetings:

… the presence of the poor is limited to pictures in slide shows while wealthy people hobnob over cocktails and abundant buffets. Am I the only one who would rather hear about what life as a poor woman in Ethiopia is like from an actual poor Ethiopian woman?

WEDNESDAY

0900 hrs: The “UN Digital Media Lounge” is where they keep the bloggers who couldn’t get real press credentials to attend the UN summit. There’s wifi, coffee and bagels, but at 47 blocks north of the actual UN building it feels a bit removed from the MDG summit. All day, different heads of state are speaking at the UN on “integrated and coordinated implementation of and follow-up to the outcomes of the major United Nations conferences and summits in the economic, social and related fields; and follow-up to the outcome of the Millennium Summit: draft resolution.” Some of these are broadcast on the screen at the lounge; I browse though others on the UN live feed site.

1430 hrs: The most hyped event of the day is the launch of a new global health strategy for child and maternal health, “Every Woman, Every Child.” Secretary General Ban Ki-Moon promised a “clear road map for making a fundamental difference in millions of lives.” Then he opened the floor to two-minute speeches from practically everybody in the room: poor countries, rich countries, foundations, corporations, NGOs, all making promises and pledges, which the UN announced amounted to “over $40 billion in resources for women and children’s health.”

Oxfam UK questions whether the funds pledged for women and children are actually additional funds or just “promises with a seemingly big price tag in a new shiny UN wrapper.”

1700 hrs: Meeting fatigue is setting in. Since I’m not invited to the MDG Gala, where attendees will celebrate pledges to fight poverty in New York’s swankiest Plaza Hotel, I’m grabbing a beer, going home, and watching President Obama’s speech from the comfort of my couch.

…Wait a minute, did President Obama really just admit the US approach to food aid is creating dependence, not development, and that our aid policies have focused on short term gains at the expense of sustainable development? Did he just become the world’s latest aid skeptic? Did he just pledge to be guided by evidence, “to invest in programs that work, and end those that don’t”? Judging from immediate reactions, people watching are starting to get that some of that old “Yes, we can” feeling.

THURSDAY

1045 hrs: It’s Raj Shah again, stopping by the UN Digital Media Lounge. Wow, did you know he’s only 37 years old, a medical doctor with a degree in health economics? The guy is impressive. But he doesn’t address the most obvious follow-up question to Obama’s speech last night: What happens next so that Obama’s hopey-changey speech gets translated into actual change in our 50-year-old aid legislation and at USAID and the 25 other government agencies involved in US foreign assistance? Will development really be elevated on par with diplomacy and defense when the White House’s new policy says that Shah will report to the Secretary of State, and will have a seat on the National Security Council only “as appropriate”?

1400 hrs: And, we’re back at CGI for a special panel on Haiti’s reconstruction. Uh-oh, is Haiti’s President René Préval really inviting Wyclef Jean on stage? President Clinton talks investment climate with the CEO of Royal Caribbean, the cruise line that brought in more than half of Haiti’s tourists last year. He describes a Coca-Cola/IDB/TechnoServe project sourcing Haitian mangos for a new Odwalla mango-lime juice, and speaks movingly about the resilience of the Haitian people.

Coca-Cola is everywhere this week, in the speeches of Melinda Gates, Raj Shah, in multiple panels at CGI. The prominence of corporations in this week’s events led to at least one wry comment about “saving the world with high-fructose corn syrup” and an observation that we’re hearing “more and more about mutual benefit and less about the moral requirement to help those in need.”

Given the overlap in timing, topics and headline speakers (Hillary Clinton, Mohammed Yunus, Ellen Johnson-Sirleaf, Melinda Gates and Pres Obama all spoke at both events), comparisons between the two events are inevitable, with one journalist suggesting the CGI could be “the new UN.” It does have far better production values, better food, and better (though still spotty) press access. Come to think of it, Bill Clinton would make a bit more inspirational SecGen than the mild-mannered one we have now. But let’s not forget that CGI members fork out $20,000 per year for the privilege to attend what is still, despite the roster of impressive accomplishments, a club for very privileged people.

1730 hrs: The leaders of the MDG summit have issued their “outcome document,” whose long stretches free of content, by custom, were agreed upon before the delegates even arrived:

We underscore the continued relevance of the outcomes of all major United Nations conferences and summits in the economic, social and related fields and the commitments contained therein, including the Millennium Development Goals….We strongly reiterate our determination to ensure the timely and full implementation of these outcomes and commitments.

Clinton is creating more of a pulse closing his show, which I’m watching from the press pen since they couldn’t fit half the press people into the mobbed closing session. Looking out at the audience (Oh my God that’s Mick Jagger!), Clinton quips that while “politics is show business for ugly people,” work in the non-governmental sector is “show businesses for nerds.” For this week, at least, he’s right, it’s been quite a spectacle. Thank goodness there’s 12 months until the next one.

Heated debate with John McArthur on MDGs and accountability

In 2000, nearly every country in the world made a promise to achieve a set of eight goals, including poverty reduction, women’s empowerment and universal primary education by 2015. How far have we gotten? Host Michel Martin speaks with two opposing voices about the progress made this far: John McArthur, CEO of Millennium Promise, and William Easterly, professor of economics at New York University.

Listen to the interview on NPR's Tell Me More. Once in the media player, the segment is called "UN Convenes to Assess Global Progress"- it's 12 minutes long.

The Millennium Development Goal that really does work has been forgotten

UPDATE 12 noon: this  is a dueling oped with Sachs on ft.com, debate has moved on and even some agreement (see end of post) from a column in the on-line Financial Times today ; for ungated access and a picture of the handsome author go here. The Millennium Development Goals tragically misused the world’s goodwill to support failed official aid approaches to global poverty and gave virtually no support to proven approaches. Economists such as Jeffrey Sachs might argue that the system can be improved by ditching bilateral aid and moving towards a “multi-donor” approach modelled on the Global Fund to Fight Aids, Tuberculosis and Malaria. But current experience and history both speak loudly that the only real engine of growth out of poverty is private business, and there is no evidence that aid fuels such growth.

Of the eight goals, only the eighth faintly recognises private business, through its call for a “non-discriminatory trading system”. This anodyne language refers to the scandal of rich countries perpetuating barriers that favour a tiny number of their businesses at the expense of impoverished millions elsewhere. Yet the trade MDG received virtually no attention from the wider campaign, has seen no action, and even its failure has received virtually no attention in the current MDG summit hoopla.

This is all the more misguided because trade-fuelled growth not only decreases poverty, but also indirectly helps all the other MDGs. Yet in the US alone, the violations of the trade goal are legion. US consumers have long paid about twice the world price for sugar because of import quotas protecting about 9,000 domestic sugar producers. The European Union is similarly guilty.

Equally egregious subsidies are handed out to US cotton producers, which flood the world market, depressing export prices. These hit the lowest-cost cotton producers in the global economy, which also happen to be some of the poorest nations on earth: Mali, Burkina Faso and Chad.

According to an Oxfam study, eliminating US cotton subsidies would “improve the welfare of over one million West African households – 10 million people – by increasing their incomes from cotton by 8 to 20 per cent”.

Brahima Outtara, a small cotton farmer in Logokourani, Burkina Faso, described the status quo to the aid agency a few years ago: “Cotton prices are too low to keep our children in school, or to buy food and pay for health.”

To be fair, the US government has occasionally tried to promote trade with poor countries, such as under the African Growth and Opportunity Act, a bipartisan effort over the last three presidents to admit African exports duty free. Sadly, however, even this demonstrates the indifference of US trade policy towards the poor.

The biggest success story was textile exports from Madagascar to the US – but the US kicked Madagascar out of the AGOA at Christmas 2009. The excuse for this tragic debacle was that Madagascar was failing to make progress on democracy; an odd excuse given the continued AGOA eligibility of Cameroon, where the dictator Paul Biya has been in power for 28 violent years. Angola, Chad and even the Democratic Republic of the Congo are also still in. The Madagascan textile industry, meanwhile, has collapsed.

In spite of all this, the great advocacy campaign for the millennium goals still ignores private business growth from trade, with a few occasional exceptions such as Oxfam. The burst of advocacy in 2005 surrounding the Group of Eight summit and the Live 8 concerts scored a success on the G8 increasing aid, but nothing on trade.

The UN has continuously engaged US private business on virtually every poverty-reducing MDG except the one on trade that would reduce poverty-increasing subsidies to US private business. And while the UN will hold a “private sector forum” on September 22 as part of the MDG summit, the website for this forum makes no mention of rich country trade protection.

The US government, for its part, announced recently its “strategy to meet the millennium development goals”. The proportion of this report devoted to the US government’s own subsidies, quotas and tariffs affecting the poor is: zero. News coverage reflects all this – using Google News to search among thousands of articles on the millennium goals over the past week, the number that mention, say, “cotton subsidies” or “sugar quotas” is so far: zero.

It is already clear that the goals will not be met by their target date of 2015. One can already predict that the ruckus accompanying this failure will be loud about aid, but mostly silent about trade. It will also be loud about the failure of state actions to promote development, but mostly silent about the lost opportunities to allow poor countries’ efficient private businesspeople to lift themselves out of poverty

UPDATE: this was a dueling piece with an oped by Sachs today on FT.com.

One of us also got a prestigious slot in the print edition of FT :>)

Surprising new agreement with Sachs, where he says:

{Bilateral aid doesn't work because it's} "largely unaccountable," "programmes are scattered among many small efforts," {and it creates mainly an} "endless spectacle of visiting dignitaries from donor countries."

Continuing disagreement with Sachs when he says:

The most exciting example {of success} is the Global Fund to Fight Aids, TB and Malaria. ...while a decade ago all three diseases were running out of control, now all are being reined in with millions of lives saved.

Jeff, could you clarify a bit what you mean saying that AIDS is "being reined in" when for every 100 people added to AIDS treatment, 250 people are newly infected with HIV?

Speaking at 12:15pm today at Stand Up NYC for MDGs

at Lincoln Center outdoors. Please come and join the other 5 people listening.

US gets a strategy to meet the Millennium Development Goals – please explain

UPDATE: UN Dispatch disagrees, we respond (see end of post). Although the eight goals that seek to reduce the global burden of hunger, poverty and disease were agreed upon by aid donors almost 10 years ago, and most of the goals come due in 2015, the world’s largest donor has never had a strategy to achieve them. Obama campaigned on the promise of making the MDGs “America’s goals,” but the first year and a half of his administration has not yet delivered on this promise.

On Friday, though, the US released a document written by USAID which declares that the US “fully embraces the MDGs” and “will put innovation, sustainability, tracking development outcomes, and mutual accountability at the heart of our approach to development, and, consequently, to the MDGs.” The administration is delivering this strategy just in time for the MDG summit in September.

I recently came across a paper by David Hulme that frames the fierce debate around the value of the MDGs nicely. According to Hulme, the viewpoints

… range from the high modernists, who take them at face value and are optimistic that they are a blueprint for the transformation of the human condition (Sachs,  2005); the strategic realists, who don’t believe the MDGs are a blueprint for action but believe they are essential to stretch ambitions and mobilise political commitment and public support (Fukuda-Parr, 2008); the critics, who see them as well-intentioned but poorly thought through – distracting attention from more appropriate targets (or nontargets) and more effective policies and actions (Clemens et al, 2007; Easterly, 2006);  through to the radical critics, who view them as a conspiracy obscuring the really important ‘millennial’ questions of growing global inequality, alternatives to capitalism and women’s empowerment (Antrobus, 2003; Eyben, 2006; Saith, 2006).

The US strategy is notable in that it is not internally coherent according to ANY of these alternative MDG world-views.

By doing a strategy at all, the US would seem to have placed itself firmly in the high modernist camp and to have rejected the critics’ view. But a curious feature of the new US strategy is its failure to mention the goals by name, or to strategize progress specifically towards any of the agreed-on indicators. From the critics’ point of view, this is certainly better than the UN Millennium Project’s strategy, which created a 449-step comprehensive strategy to reach the 8 goals and 18 targets. But with no concreteness on goals or indicators at all, one wonders what exactly was the point of the new US strategy according to the “high modernist” approach.

As far as “mobilizing political commitment,” it’s perhaps telling that the report is subtitled “Toward 2015 and Beyond,” which reads like an unintentional acknowledgement that the MDG exercise has already failed for 2015 – something on which this blog opined a year ago. But then again, we are a little confused why the US is now jumping on the MDG bandwagon when the band is starting to pack up their instruments.

Could it be that there should be a fifth Hulme category – the “PR view,” according to which the MDGs are a politically costless way for any given aid donor to create a positive image of benevolence towards the world’s poor, which is sadly unrelated to whether the goals are actually achieved?

--

UPDATE Monday August 2, 3:22 pm: Duncan Greene of Oxfam GB more optimistically reckons that this strategy shows “how quickly Shah and USAID have won back lost political ground from other government agencies;” UN Dispatch interviews Rajiv Shah;  Porter McConnell of Oxfam US says the development community is still waiting for the administration’s long-promised global development strategy.

UPDATE 2 Tuesday August 3, 11:36 am: Mark Leon Goldberg of UN Dispatch has responded to this blog post, arguing that The US MDG Strategy is More than a PR Ploy! and represents real progress:

….It was only five years ago that we had a freelancing UN ambassador [John Bolton] who thought he could get away with erasing the mere mention of the MDGs from a UN Summit…. Fast forward to 2010 and not only is the United States embracing the MDGs, but the administration has made it an organizing principal of US foreign policy.

Sure, the new US willingness to engage with the UN, and this new MDG strategy document, may represent progress in repairing the fractured relationship between the US and the UN. In the annals of UN bureaucratic infighting this does indeed sound like a revolutionary change.

But on the more important question of whether this change in orientation on the MDGs will make a real difference in the way that the US and other donors deliver development assistance to the world’s poor, we’re going to need more than just an official statement of intentions before we can conclude that real progress is being made.

If an evaluation is released on the internet and no one comments, does it make a sound?

The release of the Millennium Villages Project mid-point evaluation has so far been met with no discernable public response. Strange, since the release is billed as the “first major scientific report on progress after three years of MVP activity.” Doubly strange, since the MVP is an ambitious project that reaches into nearly all areas of its 500,000 recipients’ lives, and proposes, in scaled-up version, to completely change the architecture and delivery of aid to Africa.

So why the silence? Two possible reasons come to mind. Perhaps:

1. The evaluation doesn’t contain much that is unexpected or useful, and/or
2. No one really cares about evaluation.

We knew that the report would give the mid-point results of a longitudinal study comparing data from 300 Millennium Village families collected when the project began and again three, and five years later. (Although this is no longer the midpoint of anything, as the project has since expanded from 5 to 10 years.)

The new data give a picture of encouraging results across all sectors compared to the baseline. In Mwandama, Malawi, for example, bednet use for children under five increased from 14 percent to 60 percent and malaria prevalence for all age groups fell from 19 percent to 15 percent. Maize yields increased dramatically from .8 tons per hectare to 4.5 tons per hectare.

Such short-term results are positive in the sense that they describe real, immediate changes in the lives of thousands of very poor people. But they are not surprising given what we know about the level of resources and intensive technical expertise invested in these villages: the project doubles the size of the local economy—it is roughly equivalent to a 100 percent increase of per capita income per year (see here for calculations from Michael Clemens).

Unfortunately the results are also not that useful: Three years is too short a period to know how to interpret this dramatic increase in maize yields, for example. Is this consistent with normal variation in crop yields? Was 2006 an unusually good or bad year for maize? We don’t know.

The results also don’t help us determine whether current and future resources should be shifted away from other existing or even yet-to-be invented approaches, towards the MVP template. Will those short-term gains last beyond the timeline of the project? Can the project become self-sustaining?

Again, we don’t know, in part because not enough time has passed. Consider this anecdote from a New York Times blog series by Jeff Marlow on the Millennium Village of  Koraro, Ethiopia:

In 2005, all fertilizer was given away, leading to a significant increase in food production. Fertilizer subsidies were then progressively rolled back; by last year, only 50% of the cost was covered. For the 2009 growing season, the project tried something new: farmers were given loans for fertilizer, but they are expected to pay back the full cost plus interest when the harvest comes.

For many Koraro farmers, this is a daunting challenge. “The project used to help us with fertilizer,” says Brhana Syum…“But now it’s very expensive, and there’s no way to pay for it all.” Many farmers facing similar constraints have chosen to scale back their farms, thereby requiring less fertilizer, rather than face enormous debts…

So this particular push towards sustainability has come up against some obstacles. It may yet succeed, or it may fail. We don’t know the end of the story.

Supporters of the project argue that the individual interventions have already been proven: for example, we know that using better seeds and adding fertilizer will increase crop yield. But what the MVP says it is proving with this evaluation is the “value and feasibility of integrated community-based investments”—that is, the whole package of interventions, as well as the management systems used to deliver them. And this is precisely what the MVP does not have the data to demonstrate.

This evaluation repeats the call to scale up the project within existing project countries and expand to new ones, as quickly as possible. But the MVP as a whole remains an untested and unproven intervention, while the lives of Millennium Villagers—their habits, beliefs, livelihoods, and sources of authority—are  inevitably being changed in profound ways. This evaluation does nothing to change the argument of my previous post that the MVP should live up to their promise to be a ‘proof of concept:’ to be seriously and independently evaluated, and proven to work—beyond immediate short-term effects—before it is scaled up.

If you were sick and someone offered you a drug that hadn’t been tested, would you take it? And even if you would, would you want hundreds of millions of people whose lives depended on it to forego other types of treatment and take that drug too?

Was Africa set up to fail on the Millennium Development Goals?

“Africa far from reaching Millennium Development Goals”

News report on African Development Bank conference, May 28, 2010

“No country in sub-Saharan Africa is on course to achieve all the Goals by 2015.”

United Nations, Key Messages for September 2010 Summit

“It is easy to see that Sub-Saharan Africa lags on all the MDGs.”

World Bank and IMF, Global Monitoring Report 2010, full text download page here

Africa has had some successes and good news in the last decade, and has made much progress on some social indicators over a longer period. Why isn’t that reflected in this drumbeat of universal failure on the MDGs?  The answer is that Africa was set up (probably unintentionally) to fail on the MDGs.

This set-up to fail is not well known, probably because you have to go through some really boring details to document it. One problem with the MDGs is that success on a goal is very sensitive to how you define the goal. There are actually three different choices you have to make to define a goal in 2015 relative to the 1990 baseline. Let’s use primary enrollment as an example, and say that a random country went from 50 percent primary enrollment in 1990 to 90 percent enrollment in 2015.

(1) Do you define the goal in CHANGES or in LEVELS (e.g. the change in primary enrollment from 1990 to 2015, or the level of primary enrollment attained in 2015)?

If your answer to (1) is CHANGES, you still need to make two more decisions:

(2) Will it be PERCENT CHANGE or ABSOLUTE CHANGE (e.g. percentage change in primary enrollment (90-50)/50=80%, or the absolute change (40 percentage points)?

(3) Will you use the USUAL social indicator or its REVERSE (e.g. percent enrolled OR percent NOT enrolled)?

(1) defines two possible indicators of LEVELS vs. CHANGES, and the different combinations of (2) and (3) create 4 different ways to define CHANGES. The MDGs did not make consistent choices, but actually use 4 out of the 5 possible combinations for MDGs 1 through 7, as shown in the table (where the actual indicator used is highlighted in yellow).

These choices are not neutral. Initial conditions determine whether a given choice makes it easier or harder to meet the Goal. Most obviously, if you have a LEVEL goal (primary enrollment, gender equality), the further away you are at the beginning, the harder it is to meet the goal.  This was true for Africa for MDGs 2 and 3. (A plus sign shows whether each way of stating the goal would have INCREASED the probability of Africa making it, a MINUS sign indicates the choice of goal made it harder for Africa, relative to other countries with different initial conditions).

If you have a high initial level then a PERCENTAGE DECREASE is harder to achieve.  This was true for poverty (MDG 1)  and child mortality (MDG 4) in Africa. It was also true for MDG 7, because the REVERSE INDICATOR was used (percent WITHOUT clean water instead of the USUAL INDICATOR:  percent WITH clean water).

If you have a low initial level, on the other hand, a PERCENTAGE INCREASE would be easier to achieve. If the MDGs had been set in terms of PERCENT CHANGE in the USUAL INDICATOR (as it was for MDGs 1 and 4), then Africa would have easily met MDGs 2,3, and 7 on primary enrollment, gender equality in enrollment, and percent with clean water, and all of the above statements about universal African MDG failure would be nonsense.

Instead, each of the goal choices made for MDGs 1, 2, 3, 4, and 7 made it HARDER (if not impossible) for Africa to meet the goals, compared to alternative, equally plausible choices. ANY consistent choice of goal type except LEVELS would have made it easier for Africa to meet some of the goals.

The craziest statements implicitly made above are that Africa is failing on MDGs 5 and 6 (67% reduction in maternal mortality and reversing spread of AIDS), since there were NO data on these indicators for the benchmark year of 1990 (actually  hardly ANY reliable data for any year until very recently).

So this is SO boring, and who cares? Nobody, apparently, because the sweeping statements about Africa failing are made every year without anyone bothering to check the details. Because Africa ALWAYS fails, right?

I published an academic paper on all this here in 2009, with a working paper version available since 2007. Michael Clemens and Todd Moss and Clemens, Moss, and Charles Kenny had made related criticisms as early as 2004-2005. Michael Clemens in 2004 described how Africa's progress in education was impressive by both historical and contemporary standards. The MDG crowd are aware of these papers. For example, the World Bank and IMF quote them in their recent Global Monitoring Report 2010 . This report acknowledges “what low-income countries {mainly African} achieved before the crisis {on the poverty and social indicators}  is indeed remarkable.” But this point was soft-pedaled and has had no effect on the endlessly reiterated "Africa fails on all MDGs" line.

I think the MDG design was unintentional after some conversation with the original creators of the goals, who did not intend the MDGs to be applied at the regional or country levels. What is less forgivable is the aquiescence in making Africa look like a failure after the bias was clear to anyone who would bother to check. Of course, there are areas and time periods where Africa has done badly, but is that any reason to take the successes and make them look like failures?

Jeff Sachs’ intellectual empire gets new funding

There’s a new way to study development: a masters degree in the practice of development. The MacArthur Foundation announced ten universities to receive funding for the new degree program yesterday, bringing the funding from MacArthur for this project to $16 million. The first students matriculated at Columbia University in 2009, and by 2013 the foundation expects the programs to be producing 400 graduates a year from around the world.

The two-year degree is multidisciplinary—the health sciences, the natural sciences and engineering, the social sciences, and management—with a focus on application and fieldwork.

Since today’s problems—like climate change, poverty and sustainable development—are interconnected, students need to be prepared to think across disciplines, so the argument goes. If ending global hunger (Millennium Development Goal number one) requires technical knowledge of health and nutrition, agronomy, agricultural supply systems, as well as managing organizational change, then this degree proposes to equip graduates with basic knowledge on all those topics.

The idea for the new global program comes from the Earth Institute’s Jeffrey Sachs, an architect of the Millennium Development Goals, and John McArthur, the head of the NGO that supports the Millennium Village Project, who articulated their vision in a 2008 report on education for development professionals.

Here’s what this program assumes the world needs more of:

a new generation of development practitioners who can understand the “languages” and practices of many specialties, and who can work fluidly and flexibly across intellectual and professional disciplines and geographic regions.

This sounds pretty good. In fact, I’m a generalist myself, which is how I ended up in this job, where I write about a global health issue one day and an economics paper the next.

But what if what the world really needs more of something else? What if it needs more specialists, more people with deep knowledge about the regions they study and work in? What if it needs people who are well-versed enough in their own disciplines to be critical of half-baked development ideas cooked up by aid planners who know just enough about every topic to believe they have the answers?  What if the world needs more specialists to evaluate the quality of the work in each specialty?

Curriculum and course materials proposed by the central “Secretariat” for development practice are housed in Columbia’s Earth Institute. Will the new programs produce students with a standardized, narrowly-prescribed view of how to approach development problems? Or will the melding of disciplines encourage critical thinking and help straddle the theory-policy divide, making global cooperation run more smoothly and international aid more effective?

I hope it’s the latter. But here’s one discouraging clue: The draft 2009 syllabus for the development practice degree’s required “foundation course,” offered at Columbia and several other universities around the world through web conferencing, reads like a synopsis of the degree itself. And all the readings for the course’s introductory week, the week devoted to foreign aid and policy, and the week on the Millennium Villages Project are authored by either Jeff Sachs, John McArthur, the Millennium Villages Project scientists, or the UN.

Hat tip to Michael Clemens.

Michael Clemens won’t let up on the Millennium Villages + bonus links

It’s nice to see scholars bringing attention to the critical need for evaluation and informed public dialogue (not just “success stories” or short-term impact evaluation) for the Millennium Villages Project, which we have also covered on this blog. Michael Clemens of the Center for Global Development is currently carrying on a very revealing dialogue with Millennium Villages. In Michael’s first blog post which we blogged, he makes three central points:

1. The hundreds of thousands of people living in the Millennium Villages, present and future, deserve to know whether the project’s combination of interventions is backed up by good science.
2. Randomized evaluation is the best way to do this. While it may be too late to properly evaluate the first wave of villages, there is still time to conduct such a study for the next wave of villages.
3. The MVP evaluation should demonstrate long-term impact before it is scaled up.

In a subsequent post, Michael parses the curious non-answer he receives from the director of monitoring and evaluation for the MVP, Dr. Paul Pronyk. He breaks down—for those of us not intimately involved in the finer details of impact evaluation—the difference between true scientific evaluation and what the MVP says it is doing, namely “matched randomly selected comparison villages.”

What the MVP has done is something very different from…a rigorous evaluation.  First, village cluster A1 was chosen for treatment, for a range of reasons that may include its potential for responding positively to the project.  Then, long after treatment began, three other clusters that appear similar to A1 were identified — call these “candidate” comparison clusters A2, A3, and A4.  The fact that all three candidates were chosen after treatment in A1 began creates an enormous incentive to pick those candidates, consciously or unconsciously, whose performance will make the intervention in A1 look good.  Then the comparison village was chosen at random from among A2, A3, and A4.

Differences between the treated cluster and the comparison cluster might be due to the MVP. But those differences might also be due to how the original Millennium Village was chosen, and how the three candidate comparison villages were chosen.  This is not a hypothetical concern…

So, either the MVP director of evaluation does not understand evaluation...or he thinks we won't know the difference.

Dr. Pronyk promises the release of the MVP’s midpoint evaluation at some unspecified time later this year, and said they “look forward to an active discussion about the initial findings regarding poverty, hunger, and disease in the Millennium Villages.” We hope the scholarly community and the wider reading public concerned with development issues will give Dr. Pronyk precisely what he’s asking for.

Bonus Links

* Sounds a bit like a parody we wish we’d written….but it’s true. Yesterday’s NYT features this quote from a story on China’s bid to supply California with technology, equipment and engineers to build a high-speed railway, and to help finance its construction:

“We are the most advanced in many fields, and we are willing to share with the United States,” Zheng Jian, the chief planner and director of high-speed rail at China’s railway ministry, said.

* We’d be remiss not to mention this helpful timeline of celebrity aid to Africa featuring an interactive map from Mother Jones (and some additional commentary from Wronging Rights and Texas in Africa.)

Beware the fury of a patient man: Michael Clemens on Millennium Villages

Michael Clemens at the Center for Global Development is a very calm,  judicious, sensible guy. But even he has finally lost patience with the lack of any serious evaluation of the Millennium Villages: Why a Careful Evaluation of the Millennium Villages is Not Optional

UPDATE (3/20, 8:16am) Chris Blattman adds his take on this.

New York Times on Millennium Villages

Jeffrey Gettleman reports today from Sauri, Kenya on the debate on the Millennium Villages.

I'm really not sure what to make of this

Millennium Villages Comments, We Respond

We received the following comment this morning from the Director of Communications at the Earth Institute, regarding the Aid Watch blog published yesterday, Do Millennium Villages Work? We May Never Know. My response is below.----- It’s unfortunate that the author of this post chose to publish such an uninformed blog on the Millennium Village Project’s monitoring and evaluation activities. She and William Easterly at Aid Watch were invited to meet with our scientists and discuss the science and research behind the Villages and the details of the MVP monitoring and evaluation process before publishing any commentaries. Instead the author hastily chose to publish without talking with MVP researchers. The inaccuracy of the blogpost is a reflection of the lack of rigor and objectivity with which the Aid Watch authors approach this subject time and again.

For readers interested in reading factually accurate information about the Millennium Villages project and its monitoring and evaluation strategy, please see: http://www.millenniumvillages.org/progress/monitoring_evaluation.htm

Erin Trowbridge Director of Communications The Earth Institute

----- Dear Erin,

I had hoped for a different kind of response, one that addressed the specific points made by the piece. Your only comment on content is to say the piece was "uninformed." It would be helpful if you would clarify exactly what you think the piece got wrong, and offer what you view as the correct information to replace it. I would be happy to post such a response on Aid Watch.

Your comment is in any case an inaccurate characterization of our interaction over the past two months. I sent seven separate emails to you and one to CEO John McArthur beginning in mid-August, asking for information on the overall MV evaluation strategy, and eventually asking specifically for an explanation of how the thinking of the team had evolved from 2006 (when Jeff Sachs said there were no controls) to 2009 (when we were informed that there are comparison villages for 10 MV sites). Your responses were represented fairly in the blog post that Aid Watch published yesterday. We expressed willingness to meet with the research scientists after you offered this; it is unfortunate that we were unable to find a mutually convenient time to meet before our publication deadline, which we had already postponed several times.

Thank you for sharing further details of the MVP evaluation process with the information that has now appeared on the link you provided. Interested readers can now independently judge for themselves the merits and demerits of the ongoing MVP evaluation.

Frequent readers here may tire of hearing it, but it is our belief that greater transparency and a greater willingness on the part of donors and aid practitioners to share information with supporters and skeptics alike will make aid better.

Laura

Laura Freschi Associate Director Development Research Institute

Do Millennium Villages work? We may never know

Jeffrey Sachs’ Millennium Villages Project has to date unleashed an array of life-saving interventions in health, education, agriculture, and infrastructure in 80 villages throughout ten African countries. The goal of this project is nothing less than to “show what success looks like.” With a five-year budget of $120 million, the MVP is billed as a development experiment on a grand scale, a giant pilot project that could revolutionize the way development aid is done.

But are they a success? To address that question, we need to know: What kind of data is being collected? What kinds of questions are being asked? Three years into the start of one of the highest-profile development experiments ever, who’s watching the MVPs?

The most comprehensive evaluation of the project published so far is a review by the Overseas Development Institute, a large UK-based think tank. The review covered two out of four sectors, in four out of ten countries, with data collected in the MVs only, not in control villages. The report’s authors cautioned that “the review team was not tasked and not well placed to assess rigorously the effectiveness and efficiency of individual interventions as it was premature and beyond the means of the review.”

Despite this, a Millennium Villages blog entry on Mali says, “With existing villages showing ‘remarkable results,’ several countries have developed bold plans to scale up the successful interventions to the national level.” Millennium Promise CEO John McArthur described Sachs’ recent testimony to the Senate Foreign Relations Committee: “Sachs noted the success of the Millennium Villages throughout Africa and the tremendous development gains seen in the project over the past three years.”

The Evaluation that Isn’t?

In contrast, evaluation experts have expressed disappointment in the results they’ve seen from the Millennium Villages Project to date. This isn’t because the MVPs fail to produce impressive outcomes, like a 350 percent increase in maize production in one year (in Mwandama, Malawi), or a 51 percent reduction in malaria cases (in Koraro, Ethiopia). Rather, it has to do with what is—and is not—being measured.

“Given that they’re getting aid on the order of 100 percent of village-level income per capita,” said the Center for Global Development’s Michael Clemens in an email, “we should not be surprised to see a big effect on them right away. I am sure that any analysis would reveal short-term effects of various kinds, on various development indicators in the Millennium Village.” The more important test would be to see if those effects are still there—compared with non-Millennium Villages—a few years after the project is over.

Ted Miguel, head of the Center of Evaluation for Global Action at Berkeley, also said he would “hope to see a randomized impact evaluation, as the obvious, most scientifically rigorous approach, and one that is by now a standard part of the toolkit of most development economists. At a minimum I would have liked to see some sort of comparison group of nearby villages not directly affected by MVP but still subject to any relevant local economic/political ‘shocks,’ or use in a difference-in-differences analysis.” Miguel said: “It is particularly disappointing because such strong claims have been made in the press about the ’success’ of the MVP model even though they haven't generated the rigorous evidence needed to really assess if this is in fact the case.”

An MVP spokesperson told me that they are running a multi-stage household study building on detailed baseline data, the first results from which will be published in 2010. The sample size is 300 households from each of the 14 MV “clusters” of villages (which comprise about 30,000-60,000 people each.) She also said that their evaluation “uses a pair-matched community intervention trial design” and “comparison villages for 10 MV sites.”

But Jeff Sachs noted in a 2006 speech that they were not doing detailed surveying in non-MV sites because—he said— “it’s almost impossible—and ethically not possible—to do an intensive intervention of measurement without interventions of actual process.” A paper the following year went on to explain that not only is there no selection of control villages (randomized or otherwise), there is also no attempt to select interventions for each village randomly in order to isolate the effects of specific interventions, or of certain sequences or combinations of interventions.

CEO John McArthur declined to comment on this apparent contradiction. The MVP spokesperson could say only that the evaluation strategy has evolved, and promised a thorough review of their monitoring and evaluation practices in 2010.

Comparison villages could be selected retroactively, but the MVP has failed to satisfactorily explain how they chose the MVs, saying in documents and in response to our questions only that they were “impoverished hunger hotspots” chosen “in consultation with the national and local governments.” If there was no consistent method used in selecting the original villages (if politics played a role, or if villages were chosen because they were considered more likely to succeed), it would be difficult to choose meaningful comparison villages.

Living in a Resource-Limited World

Imagine that you are a policymaker in a developing country, with limited resources at your disposal. What can you learn from the Millennium Villages? So far, not very much. Evaluations from the MVP give us a picture of how life has changed for the people living in the Millennium Villages, and information about how to best manage and implement the MVP.

Sandra Sequeira, an evaluation expert at London School of Economics, sums up the quandary neatly. “Their premise is that more is always better, i.e. more schools, more clinics, more immunizations, more bed nets. But we don't live in a world of unlimited resources. So the questions we really need to answer are: How much more? Given that we have to make choices, more of what?”

These are tough questions that the Millennium Villages Project will leave unanswered. For a huge pilot project with so much money and support behind it, and one that specifically aims to be exemplary (to “show what success looks like”), this is a disappointment, and a wasted opportunity.