Reasons to doubt new health aid study on fungibility

This post is by David Roodman, a research fellow at the Center for Global Development (CGD) in Washington, DC. A couple of weeks ago, researchers at the Institute for Health Metrics and Evaluation triggered a Richter-7 media quake with the release of a new study in the Lancet.

Here’s how the Washington Post cast the findings:

After getting millions of dollars to fight AIDS, some African countries responded by slashing their health budgets.

Laura Freschi at Aid Watch blogged it too.

I am not a global health policy wonk, and I don’t play one on this blog, but it may well be the case that I wrote the program that produced the headline numbers (for every dollar donors gave to governments to spend on health, governments cut their own spending by $0.43–1.17).

I find the results generally plausible. I also don’t particularly believe them. Let me explain.

The results are plausible because it is easy to imagine that health aid is partly fungible: governments can take advantage of outside finance for health by shifting their own budget to guns, gyms, and schools. I would. Wouldn’t you? Well, maybe except for the guns part.

The results are dubious because it is an extremely hairy business to infer causation from correlations in cross-country data. That’s why Bill once sighed about the:

1 millionth attempt to resolve the relationship in a cross-country growth regression literature that is now largely discredited in academia.

The variable being explained here is not growth but recipient governments’ aid spending, which is admittedly less mysterious. But skepticism is still warranted. Consider:

• The model may be wrong. The study assumes that aid received in a year only directly affects government spending that same year, even though it could take longer for the money to pipeline through—especially if recipients bank the aid to smooth its notorious volatility (hat tip to Mead Over; also see Ooms et al. Lancet commentary).
• The quantities of interest are health-aid-to-governments and government-health-spending-from-own-resources, which is calculated as total government-health-spending minus health-aid-to-governments (yes, the variable I just mentioned above). So if health-aid-to-governments were systematically overestimated for some countries and years, government-health-spending-from-own-resources would automatically be underestimated.For example, suppose the study is wrong, that there is no relationship between health aid and governments’ health spending from their own resources. Suppose too that health aid to some countries, as measured, includes payments to expensive western consultants. That money would never reach the receiving government, resulting in an overestimate of actual aid receipts and an underestimate of how much governments are contributing to their own health budgets. The analysis would then spuriously show higher health aid causing governments to slash their own health spending. In another Lancet commentary, Sridhar and Woods list four possible sources of mismeasurement of this sort.

Both these problems must be present to some extent, creating mirages of fungibility.

Understanding at least the latter problem of causality, the authors feed their data into a black box called “System GMM.” (They call it “ABBB,” using the initials of the people who invented it.) I am in an intimate, long-term relationship with System GMM, having implemented it in a popular computer program. I have worked to demystify System GMM and documented how, just by accepting standard default choices in running the program, you can easily fail to prove causality while appearing to succeed. I can’t explain why without getting technical, which is not to say that only I know the problem – it is very well known among economists with some minimum econometric competence – but NOT to everyone who actually uses the techniques. Suffice it to say that I sometimes feel like this black box is a small time bomb that I have left ticking on the landscape of applied statistical work.

Responsible use of this black box involves telling your readers how you set all the switches and dials on it, as well as running certain statistical tests of validity. The Lancet writers have not done these things (yet). Nor have they shared their full data set. So it is impossible to judge how well their claims about cause and effect are rooted in the data. If replicability is a sine qua non of science, then this study is not yet science.

Does health aid to governments make governments spend more on health?

If you’re not an economist, you might reasonably assume that the answer to this question is yes. The story might go something like this: aid agencies give money to poor country governments to distribute bed nets or give vaccinations, and those additional funds are added to whatever money the country was able to scrape together to spend on health before the donor came along. As a result of the health aid, the total amount of money spent on health increases. There is new evidence, from a study from the Institute for Health Metrics and Evaluation published in the Lancet last week, showing that this story doesn’t describe what’s really going on. Overall, global public health financing shot up by 100 percent over the last decade, but the study’s authors found that on average, for every health aid dollar given, developing country government shifted between $.43 and $1.17 of their own resources away from health. The trend is most pronounced in Africa, which received the largest amount of health aid.

The finding that health aid substitutes for rather than complements existing government health spending has caused a mini- scandal in the press precisely because it runs so counter to people’s optimistic expectations, perpetuated by aid agencies’ fund-raising campaigns, about the level of control that donors can exert over the spending of developing country governments.

Economists, on the other hand, have been beating the dismal drum for a long time on this issue. In 1947, Paul Rosenstein-Rodin, then a deputy director at the World Bank, famously said, “When the World Bank thinks it is financing an electric power station, it is really financing a brothel.” Economists expect that aid will be at least partially fungible (that is, that aid money intended by donors for one sector or project can and will be used by governments interchangeably with funding for other priorities), and this prediction is borne out by empirical studies from the late 1980s on. The authors of a 2007 paper in the Journal of Development Economics observed, “While most economists assume that aid is fungible, most aid donors behave as if it is not.”

You might argue (as Owen Barder does in depth here) that recipient governments are acting rationally in response to erratic donor funding, which ebbs and flows according to donor priorities and how well the global community mobilizes fundraising around a particular issue in any given year. After all, doesn’t the donor community’s insistence on country ownership mean that they want poor country governments to be able to set their own budget priorities?

The problem is that aid agencies have long used the argument that earmarking aid for a specific project or sector is a credible way to force recalcitrant recipient country priorities into line with donor priorities—to coerce bad governments into making good decisions.

If  governments that don't prioritize their people's welfare respond to an influx of aid money by simply shifting their existing resources around to circumvent donor priorities (and we don’t know what is happening to the resources shifted away from health—they could be going to private jets and presidential palaces, or to education, infrastructure, or loan repayments, or really anything at all),  then the aid agency argument for project aid falls apart. The burden of proof correctly lies with the aid agencies to show that aid isn’t freeing up funds for bad governments to use badly.

The Lancet findings are scandalous, relative to the naïve but widespread belief that donors can use earmarked aid to force bad governments to behave.