Controlled experiments and uncontrollable humans

Bill reviewed two much-awaited books for the Wall Street Journal last weekend: Poor Economics by Abhijit Banerjee and Esther Duflo, and More Than Good Intentions by Dean Karlan and Jacob Appel. The Good:

The books' signal achievement is in addressing two disgraceful problems that beset humanitarian aid. The first is that the effectiveness of aid is often not evaluated at all; the second is that even when aid is evaluated, the methods are often dubious, such as before-and-after analysis that doesn't take into account variables that have nothing to do with the aid itself. Humanitarian aid is usually flying blind. These books take the blinders off—de-worming does work, many other efforts do not.

But things are not as simple as they first appear. The authors are brutally honest about how difficult poverty-alleviation is....

...

In addition to testing out ideas, such field work also has the benefit of letting researchers chat informally with poor people—conversation that can be thoroughly illuminating. What looks like irrationality may just be the failure of outsiders to fully appreciate the problem...

....

“More Than Good Intentions” and “Poor Economics” are marked by their deep appreciation of the precariousness that colors the lives of poor people as they tiptoe along the margin of survival. But I would give an edge to Mr. Banerjee and Ms. Duflo in this area—the sheer detail and warm sympathy on display reflects a true appreciation of the challenges their subjects face. Messrs. Karlan and Appel are at their best in addressing the subtleties of behavior and testing them in the psychology laboratory and in the field. They have produced a remarkably readable and credible analysis of the intertwining of irrationality and poverty.

The Not-so-Good:

Unfortunately, the books also indulge another sort of irrationality: the demand for big, general statements even if you’re discussing limited, context-specific matters. The authors criticize over-generalizing and over-promising in the aid business, but they too often do their own exaggerating when it comes to what their methods can deliver. Both books end with overselling, “five key lessons” (Banerjee and Duflo) or “seven ideas that work” (Karlan and Appel), overriding their own previous cautions about sensitivity to context and the limits to each intervention. Other economists criticize overselling as a common fault of those who do these small experiments.

Read the full review (ungated) here.