Is Impact Measurement a Dead End?

This post was written by Alanna Shaikh. Alanna is a global health professional who blogs at UN Dispatch and Blood and Milk. We’ve spent the last few years watching the best donors and NGOs get more and more committed to the idea of measurable impacts. At first, the trend seemed unimpeachable. International donors have spent far too much money with far too few results. Focusing more on impact seemed like the way out of that trap.

But is it? The last couple of weeks have seen a spate of arguments from development thinkers rethinking this premise.

Steve Lawry at the Hauser Center, argues two main points against excessive focus on impact evaluation. The first is that it stifles innovation by keeping NGOs from trying risky new things. But I think that the problem is an institutional culture that doesn’t allow for failure. By allowing NGOs to fail and learn from failure, innovation is encouraged.

His second point is more interesting: “Many real-world problems are not easily described with the kind of precision that professional mathematicians insist upon. This is due to the limitations of data, the costs of collecting and analyzing data, and the inherent difficulties of giving mathematical expression to the complexity of human behavior.” This strikes me as very true. At what point are we expecting too much from our impact assessments?

In the same vein, the fascinating Wanderlust blog just ran a post about Cynefin. Cynefin is a framework for understanding systems. It categorizes systems into four subsets: Simple, Complicated, Complex or Chaotic. Chaotic systems, the author argues, can’t be evaluated for impact using standard measures. He states that “In a Chaotic paradigm, there is relatively little difference likely to occur in quality between a response that is based on three weeks’ worth of detailed analysis and one that is based on the gut reaction of a team leader…”

The Center for Global Development just published a paper by former USAID administrator Andrew Natsios. Natsios points out that USAID has begun to favor health programs over democracy strengthening or governance programs because health programs can be more easily measured for impact. Rule of law efforts, on the other hand, are vital to development but hard to measure and therefore get less funding.

Now we come to the hard questions:

If we limit all of our development projects to those that have easy metrics for success, we lose a lot of programs, many of which support important things like rule of law. Of course, if they don't have useful metrics, how do we know those programs are supporting the important goals?

And how meaningful is impact evaluation anyway when you consider the short time frames we’re working with? Most development programs take ten years or more to show real impact. How are we supposed to bring that in line with government funding cycles?

On the other hand, we don't have a lot of alternatives to impact evaluation. Impact is not unimportant just because it’s hard to quantify at times. We can’t wish that away. Plenty of beautifully designed and carefully implemented projects turned out not to have any effect at all. For example, consider what we’ve learned from microfinance impact evaluations. Microloans have a positive effect but not the one we expected.

It’s a standard trope of this blog to point out that there’s no panacea in global development. That’s true of impact evaluation, too. It’s a tool for identifying worthwhile development efforts, but it is not the only tool.  We can’t go back to assuming that good intentions lead to good results, but there must be room for judgment and experience in with the quantifiable data.

--

UPDATE: This post was edited to correct an attribution error in the third paragraph - Eds.