5 Ways to Improve Your Impact Evaluation

Impact evaluations are supposed to tell us what works in development, and a lot of time and money goes into them. It's unfortunate, then, when they fail to report their results clearly. One of the things I found most shocking, looking through a large database of impact evaluations, was how often academic papers omitted information that is critical for interpreting the study's results and figuring out how well they might apply to other contexts. This blog post draws on data from over 400 studies that AidGrade found in the course of its meta-analyses. Here are five embarrassing things many papers neglect to report:

1) Attrition

It's normal for some people to drop out of a study. It can pose a problem, however, if attrition is not equal between the treatment group and the control group, as this self-selection process could bias the study's results. While attrition is very well-known to be something one ought report, only about 75% of papers reported it.

2) The standard deviation of key variables

Without knowing how much variation there is in an outcome variable, it's hard to know whether a paper found a relatively high or relatively low effect. Why? Often studies report outcomes that use scales particular to the paper, for example, reporting scores on a certain academic test. There is no way to compare these results across different papers using different tests unless you standardize the data - then you can at least say that program A was found to affect test scores by 0.1 standard deviations, while program B found an effect size of 0.2 standard deviations.

3) Whether the results include people who did not take advantage of the program

Intent-to-treat (ITT) estimates consider an intervention's effects on everyone assigned to receive treatment, regardless of whether or not they actually took advantage of the program. The alternative is to estimate the treatment effect on the treated (TOT). For example, suppose that only 10% of people who were offered a bed net used it, and suppose bed nets were 90% effective at preventing malaria. The TOT estimate would be 90% - the ITT estimate, 9%. Clearly, if the authors don’t take care to explain which they are reporting, we really don’t know how to interpret the results!

4) Characteristics of the context of the intervention

Are the people in your study rich or poor? It could affect how well they respond to a cash transfer. Does your intervention aim to decrease an infectious disease? It probably matters what the underlying infection rate is within the population, especially if people can catch it from each other. When did the intervention start and end relative to data collection? It is difficult to know what results mean without knowing much about the people in the study, and it makes comparing results across different settings even more difficult.

5) Comparable outcome variables

Finally, papers seem to "run away from each other" in terms of which outcome variables they cover. If one paper addresses the effect of HIV/AIDS education on the incidence of the disease, another will focus on whether people got tested. It makes sense given the incentives of the researchers to be the first ones to show a particular result and to differentiate their findings. However, a single paper's result cannot tell us how general it is. For that, you need more studies, and in order to compare those studies, they need to have outcome variables that are as comparable as possible.

Better reporting is not an impossible problem to solve. The Experiments in Governance and Politics network (EGAP), for example, decided to fund projects clustered around comparable intervention and outcome measures. In psychology, it was journals that started demanding better reporting. Something similar should happen in economics to provide researchers with the right incentives to maximize the use of their studies.

Is there rigorous evidence that anyone will use rigorous evidence?

A new study using 1,419 micro-finance NGOs as subjects created two treatment groups.

In the positive treatment subjects were randomly assigned to receive a summary of a study by prominent authors finding that microcredit is effective. The negative treatment provided information on research – by the same authors using a very similar design – reporting the ineffectiveness of microcredit. We compare both conditions to a control in which no studies were cited.

The results:

The positive treatment elicited twice as many responses as the negative treatment – and significantly more acceptances of our invitation to consider partnering on an evaluation of their program.

The authors point out that the Randomized Evaluation revolution has a missing step:

the willingness of organizations to update based on scientific information has been assumed and not established.

Or as Lant Pritchett suggested in a paper a decade ago: "It Pays to Be Ignorant"

UPDATE: see Chris Blattman take on the same paper today here.

Do free and competitive elections make a democracy? Maybe not

By Lauren Bishop Tanzania looks an awful lot like a democracy. The East African nation has been holding multi-party elections since 1995, which international observers have deemed free and competitive. In Tanzania, votes are not miscounted, opposition parties compete actively, and the ruling party—the Chama Cha Mapinduzi (CCM), which has controlled the government since independence—seems to play by the rules.

But according to Bruce Bueno de Mesquita, NYU politics professor and DRI affiliated faculty member, Tanzania is in fact sliding down a slippery slope to autocracy, even as it maintains the trappings of a “transitioning” democracy. A working paper with Alastair Smith describes how Tanzania’s completely legal and institutionalized electoral laws are placing power in the hands of a small and increasingly entrenched elite.

During Tanzania’s transition to democracy, the ruling party wrote a constitution that gave itself a significant advantage. Tanzania uses first-past-the-post plurality voting, which tends to result in two major political parties—as in both the US and the UK. But in Tanzania, the government encourages the survival of many parties, offering subsidies to presidential and parliamentary candidates. This is our first eyebrow-raising clue that something is amiss.

This artificial proliferation of political parties, along with a complicated system of direct and indirect voting, results in a Parliament where the CCM needs only one-third of the seats to reach a majority. The indirectly elected seats must be filled by women, which makes Tanzania appear progressive, but because most of the women owe their jobs to the CCM, they act as loyal rubber-stamps for the party. Raise your other eyebrow now.

Since the electoral structure allows the CCM to gain much of its support through indirect votes and appointed seats, Bueno de Mesquita and Smith found that the number of direct votes required to win a district is a powerful negative predictor of whether the ruling party will bestow goods like roads and subsidy programs on that district. Even programs specifically intended to alleviate poverty are subject to this logic: Doubling the number of direct votes required to win a district would result in a 69 percent decrease in the chance of receiving vouchers for subsidized maize. What’s even worse is that poor, more crowded districts tend to require fewer direct votes than richer, sparsely populated ones, so poor areas get fewer subsidies, and rich areas get more roads.

Tanzania receives nearly 3 billion dollars in aid each year. How many of these aid dollars have helped build roads where they’re not needed and buy cheap maize for the rich? Donors beware: you may be bankrolling the ruling party’s position of power.

-

Lauren Bishop is Online Projects Assistant at DRI, Economics Program Assistant at NYU Africa House, and an NYU MA student in International Relations.

35 Words

We hold these truths to be self-evident, that all Men are created equal, that they are endowed by their Creator with certain unalienable rights, that among these are Life, Liberty, and the Pursuit of Happiness.

The Development of Freedom since 1776 has been about including more and more men and women in that word "all".

How much do Europeans account for economic development?

From the Wall Street Journal, by Daniel Lippman:

European settlement had a longstanding positive effect on economic development in countries that were colonies, notwithstanding the terrible effects of Western diseases and political oppression that often resulted, according to new research.

The paper, titled “The European Origins of Economic Development,” was written by New York University’s William Easterly and UC Berkeley’s Ross Levine, who set out to build a new comprehensive database of the European share of the population in the early phases of colonization. It also looked at the impact of the settlers on the former colonies’ economic development today.

In an “illustrative exercise” that the two professors run in their paper, they find that “47% of average global development levels today are attributable to Europeans.”

What could accounts for that large number? The paper argues that it could partly be explained because “Europeans brought growth-promoting characteristics — such as institutions, human capital, connections with international markets, and cultural norms — that diffused to the rest of the population over generations.”

A large number of commentators generously congratulated the authors on being obvious, wrong, and racist.

You may find the NBER link to the paper above to be restricted. If so, here is an unrestricted link.

Some Not Entirely Typical Remarks by a World Bank President

The following quotations are taken from: Jim Yong Kim, Joyce V. Millen, Alec Irwin, and John Gershman, Editors, Dying for Growth: Global Inequality and the Health of the Poor, Common Courage Press: Monroe, Maine, 2000.

Introduction: What is Growing? Who is Dying? By Joyce V. Millen, Alec Irwin, and Jim Yong Kim

“This book seeks to fill an important gap in knowledge by examining the documentable health effects of economic development policies and strategies promoted by the governments of wealthy countries and by international agencies such as the World Bank, the International Monetary Fund (IMF), and the World Trade Organization.” (p. 6)

“The studies in this book present evidence that the quest for growth in GDP and corporate profits has in fact worsened the lives of millions of women and men.” (p.7)

“Even where neoliberal policy measures have succeeded in stimulating economic growth, growth’s benefits have not gone to those living in “dire poverty,” one-fourth of the world’s population.” (p. 7)

“Using Cuba as an example, Chapter Thirteen makes the case that when leaders prioritize social equity and the fundamental right of all citizens to health care, even economically strapped governments can achieve improved and more equitable health outcomes.” (p. 10)

Conclusion: Pessimism of the Intellect, Optimism of the Will, By Joyce V. Millen, Alec Irwin, and Jim Yong Kim

“Through a series of specific cases, we have demonstrated how growth – the market-led economic growth sought by governments, the growth in profits celebrated by businesses, and the growth in power and influence of transnational financial and corporate interests – often comes at the expense of the disenfranchised and vulnerable…  As the imperatives of growth at any cost increasingly determine economic and social policy and the behavior of global corporations, more people join the ranks of the poor and greater numbers suffer and die.” (p. 363)

“Today, Chomsky notes, we see widespread ‘efforts to make people feel helpless, as if there is some kind of mysterious economic law that forces things to happen in a particular way, like the law of gravitation.’ Yet belief in such an immutable law is simply ‘nonsense.’  ‘These are all human institutions, they are subject to human will, and they can be eliminated like other tyrannical institutions have been.’” (p. 390, single quote marks note quotes from Chomsky)

PUBLICATION: Dignity and Development

What do dignity and liberty have to do with institutional outcomes on development? DRI Post-doctoral Fellow Claudia Williamson publishes in the Journal of  Socio-Economics:

This paper explores how expanding the notion of informal institutions in the broader institutional framework provides a more complete explanation for development. Specifically, I incorporate McCloskey’s notion of ‘dignity and liberty’ as part of the institutional nexus. By doing so, a richer explanation and understanding of the importance of institutions in explaining different economic outcomes is offered. Focusing on bourgeois dignity offers a precise mechanism to explain how institutions matter to support economic growth. In addition, analyzing the changing attitudes towards the bourgeoisie provides a specific example of mechanisms that can lead to institutional change.

She concludes:

As such, we arrive at two main conclusions: 1) dignity must be grounded in liberty, and 2) economic exchange as a result of underlying liberties leads to dignifying the bourgeoisie. This suggests that by moving towards freer trade, poor countries today will reap the traditional benefits of free trade, but it might also set in motion a cycle for more liberties and institutional change.

WORKING PAPER: Foreign Aid's Amplification Effect on Political Institutions

Does foreign aid encourage democracy in recipient countries, or does it strengthen their dictators instead? These have long been the contradictory predictions of two competing hypotheses in the aid debate, but a new DRI working paper by Nabamita Dutta, Peter T. Leeson and DRI Postdoctoral Fellow Claudia Williamson suggests that in reality neither prediction captures the entire picture:

This paper offers a third hypothesis about how aid affects recipients’ political institutions that we call the “amplification effect.” We argue that foreign aid has neither the power to make dictatorships more democratic nor to make democracies more dictatorial. It only amplifies recipients’ existing political institutions. We investigate this hypothesis using panel data for 124 countries between 1960 and 2009. Our findings support the amplification effect. Aid strengthens democracy in already democratic countries and dictatorship in already dictatorial regimes. It doesn’t alter the trajectory of recipients’ political institutions.

The amplification effect of aid means, the authors suggest, that both competing hypotheses ascribe too much power to foreign assistance. Aid doesn't alter the institutional trajectory of any country; it makes democracies more democratic, and autocracies more dictatorial. So aid given for the "purposes of democratizing the dictatorial developing world may not only fail,"  they write, "but may actually cause harm."

What could this mean for the real purpose of aid -- economic growth and development?

To the extent that because of their stronger constraints on executive power, democracies tend to pursue better economic policies than dictatorships, when democracies receive foreign aid they become more democratic, leading to the adoption of better policies, which in turn leads to higher economic growth. Conversely, when dictatorships receive aid they become more dictatorial, preventing the adoption of better policies, which in turn prevents increases in economic growth.

Read the paper.

WORKING PAPER: Ethnicity and Conflict: An Empirical Study

Everyone has heard that ethnic divisions can lead to conflict, but is this empirically true? Moreover, how exactly might ethnicity matter during conflict? A new DRI Working Paper by Joan Esteban, Laura Mayoral and DRI Affiliated Faculty Debraj Ray study the issue in past conflicts across three measures: ethnic polarization, ethnic fractionalization and the level of cohesion within the group. From the abstract:

This paper examines the impact of ethnic divisions on con ict. The empirical specication is informed by a theoretical model of conflict (Esteban and Ray, 2011) in which equilibrium conflict is related to just three distributional indices of diversity: ethnic polarization, ethnic fractionalization, and a Greenberg-Gini index constructed across ethnic groups. Our empirical findings verify that these distributional measures are significant correlates of conflict. The underlying theory permits us to use these results to make inferences about the relative importance of public goods in conflict, as well as the extent of within-group cohesion in conflictual activity. These effects are further strengthened as we introduce country-specic measures of group cohesion and the relative importance of public goods, and combine them with the distributional measures exactly as specied by the theory.

They find that all three specifications matter for conflict:

The main result of this paper is that polarization | using linguistic distances | has a large and highly signicant impact on conflict across a number of different specifications. By and large, though with somewhat lesser consistency, this is also true of fractionalization. These two findings suggest that public and private components of conflict are generally both present, and that group cohesion is present during conflict. The numerical effects of the two measures are also quite similar and strong. For instance, moving polarization from the 20th percentile to the 80th percentile, holding all other variables at their means, approximately doubles the chances of conflict, and the same is true of fractionalization.

Read the paper.

Read about the theoretical model of conflict.

PUBLICATIONS: The Climate and Civil War Relationship

A series of recent papers by Marshall Burke, John Dykema, David Lobell, Edward Miguel and DRI Adjunct Faculty Shanker Satyanath documented evidence that warming temperatures caused by climate change increase the risk of civil war in Africa. The authors first established this connection in a published 2009 study, finding that warmer-than-average temperatures were linked to large increases in civil war in Africa between 1981 - 2002, with grave implications for the future:

We find strong historical linkages between civil war and temperature in Africa, with warmer years leading to significant increases in the likelihood of war. When combined with climate model projections of future temperature trends, this historical response to temperature suggests a roughly 54% increase in armed conflict incidence by 2030, or an additional 393,000 battle deaths if future wars are as deadly as recent wars. Our results suggest an urgent need to reform African governments’ and foreign aid donors’ policies to deal with rising temperatures.

Their results were challenged by Buhag (2010), who argued the findings were not robust as they relied on an "unorthodox" understanding of civil war, on "methodological fixes" and on the specific time period in question. In both an open letter to the Proceedings of the National Academy of Sciences, and in a new DRI working paper, Burke et al. dispute this assertion:

Buhaug’s dubious econometric choices in many of his specifications – including his decision not to control for country fixed effects or deal adequately with time trends in many specifications, or his willingness include endogenous regressors that bias all of his coefficients – further call into question his results. This of course does not imply that climate is solely “to blame” for African civil wars, as Buhaug’s provocative title would suggest that we are arguing. Rather it implies that during a particularly violent recent period in African history, variation in climate was a significant contributor to the incidence of large, destructive civil wars. We believe that this relationship is both robust and of significant interest to policy-makers tasked with reducing the incidence or impact of future conflicts.

The relationship between civil war and climate change still holds, in other words, even with alternatives measures. The authors do note, however, that this relationship has been considerably weaker in the past decade, a period of unprecedentedly high African growth and relative peace.

Read the original study on conflict and climate change here.

Read the letter to the Proceedings of the National Academy of Sciences here.

Read the latest working paper here.

WORKING PAPER: Understanding Transitory Rainfall Shocks, Economic Growth and Civil Conflict

Leaving aside these data and econometric issues, Ciccone’s surprising results do not survive obvious robustness checks.

Edward Miguel and DRI Affiliated Faculty Shankar Satyanath rebut Antonio Ciccone's (2010) assertion  that higher rainfall levels are, in fact, linked to more conflict -- a rejection of the Miguel, Satyanath and Serengeti (2004) conclusion that higher rainfall is associated with less conflict and more economic growth. But Ciccone's methods might have had some very fundamental errors:

Miguel, Satyanath and Sergenti (2004) use rainfall variation as an instrument to show that economic growth is negatively related to civil conflict in sub-Saharan Africa. In the reduced form regression they find that higher rainfall is associated with less conflict. Ciccone (2010) claims that this conclusion is ‘erroneous’ and argues that higher rainfall levels are actually linked to more conflict. In this paper we show that the results in Ciccone’s paper are based on incorrect STATA code, outdated conflict data, a weak first stage regression and a questionable application of the GMM estimator. Leaving aside these data and econometric issues, Ciccone’s surprising results do not survive obvious robustness checks. We therefore conclude that Ciccone’s main claims are largely incorrect and reconfirm the original result by Miguel, Satyanath and Sergenti (2004), finding that adverse economic growth shocks, driven by falling rainfall, increases the likelihood of civil conflict in sub-Saharan Africa.

Read the paper.

PUBLICATION: The Rise of Global Governance Indicators

A new paper by DRI affiliated faculty member Kevin Davis and NYU Law colleagues Benedict Kinsbury and Sally Engle Merry takes on the proliferation of global governance indicators like the UN Human Development index and World Bank Ease of Doing Business rankings:

The burgeoning production and use of indicators has not been accompanied by systematic study of and reflection on the implications, possibilities and pitfalls of this practice. As a result, little attention has been paid to questions such as: What social processes surround the creation and use of indicators? How do the conditions of production influence the kinds of knowledge that indicators provide? How does the use of indicators in global governance change the nature of decision-making? How does it affect the distribution of power among and between those who govern and those who are governed?

Theoretically, the authors hypothesize, indicators benefit global decision-makers. First, because the indicators simplify, they reduce the cost of decision-making. Second, decisions based on indicators are expected to be transparent, consistent, and grounded in scientific expertise, which makes the decision appear more authoritative.

But such authority may not always be warranted: the editing process required to turn raw data into easily digestible indicators removes uncertainty and makes the information seem more robust than it is.  At the same time, indicators may deepen existing inequalities, since the creators of indicators are usually powerful global actors from rich countries, and contesting indicators requires deep technical expertise.

The authors conclude:

The rapid growth in the production, use, and influence of indicators in global governance has had effects on forms of decision-making and on the shaping of public knowledge. This technology of global governance can affect the relative power and the identities of those who govern and those who are governed, and can alter patterns and possibilities of accountability. To what extent the reliance on indicators increases transparency and public scrutiny and to what extent it narrows the production of public knowledge to a small elite circle who create indicators are among key questions, with considerable theoretical and policy significance, that require substantial further empirical investigation.

The paper, "Indicators as a Technology of Global Governance" is forthcoming from the Law & Society Review. Ungated version here.

WORKING PAPER: Reestablishing the Income-Democracy Nexus

...[T]he bulk of the evidence favors of a statistical relationship between income and democratization.

Fighting words from DRI Affiliated Faculty Jess Benhabib and co-authors Alejandro Corvalan and Mark Spiegel, who use new data to overturn previous studies (Acemoglu, et al (2001), Easterly and Levine (2003), Rodrik et al (2004)) that showed good democratic institutions cause economic growth, not the other way around.

In this paper, we reexamine the robustness of the income-democracy relationship. We extend the research on this topic in two dimensions: first, we make use of newer income data, which allows for the construction of larger samples with more within-country observations. Second, we concentrate on panel estimation methods that explicitly allow for the fact that the primary measures of democracy are censored with substantial mass at the boundaries, or binary censored variables. Our results show that when one uses both the new income data available and a properly non linear estimator, a statistically significant positive income-democracy relationship is robust to the inclusion of country fixed effects.

Read the paper.

PUBLICATIONS: Rhetoric vs. Reality: The Best and Worst of Aid Agency Practices

By William Easterly and Claudia WilliamsonWorld Development, Vol. 39, No. 11, pp. 1930-1949 October 2011

Foreign aid critics, supporters, recipients, and donors have produced eloquent rhetoric on the need for better aid practices—has this translated into reality? This paper attempts to monitor the best and worst of aid practices among bilateral, multilateral, and UN agencies. We create aid practice measures based on aid transparency, specialization, selectivity, ineffective aid channels, and overhead costs. We rate donor agencies from best to worst on aid practices. We find that the UK does well among bilateral agencies, the US is below average, and Scandinavian donors do surprisingly poorly. The biggest difference is between the UN agencies, who mostly rank in the bottom half of donors, and everyone else. Average performance of all agencies on transparency, fragmentation, and selectivity is still very poor. The paper also assesses trends in best practices over time—we find modest improvement in transparency and more in moving away from ineffective channels. However, we find no evidence of improvements (and partial evidence of worsening) in specialization, fragmentation, and selectivity, despite escalating rhetoric to the contrary.

Download the paper

Download the accompanying dataset

PUBLICATIONS: Benevolent Autocrats

By William EasterlyDRI Working Paper No. 75 May 2011

Benevolent autocrats are leaders in non-democratic polities who receive credit for high growth. This paper asks two questions: (1) do theory and evidence support the concept of “benevolent autocrats”? (2) Regardless of the answer to (1), why is the “benevolent autocrats” story so popular? This paper’s answer to (1) is no. Most theories of autocracy portray it as a system of strategic interactions rather than simply the unconstrained preferences of the leader.  The principal evidence for benevolent vs. malevolent autocrats is the higher variance of growth under autocracy than under democracy. However, the variance of growth within the terms of leaders swamps the variance across leaders, and more so under autocracy than under democracy. The empirical variance of growth literature has identified many correlates of autocracy as equally plausible determinants of high growth variance. The growth effects of exogenous leader transitions under autocracy are too small and temporary to provide much support for benevolent autocrats. This paper addresses question (2) by analyzing the political economy of development ideas that makes benevolent autocrats a politically convenient concept. It also identifies cognitive biases that would tend to bias perceptions in favor of benevolent autocrats. The answers to (2) do not logically disqualify the benevolent autocrats story, but combined with (1) they suggest much greater skepticism about many claims for benevolent autocrats.

>>Download the paper, “Benevolent Autocrats”

PUBLICATIONS: Democratic Accountability in Development: The Double Standard

By William EasterlySocial Research, Vol. 77, No. 4, pp. 1075-1104 Winter 2010

The Development Establishment today tolerates a shocking double standard on democracy for the rich versus democracy for the poor. Despite both the moral and pragmatic argument for democratic rights for all, development policy discussions give little emphasis to rights for the poor. Worse, influential Western policymakers and thought leaders sing praises of autocrats such as Meles Zenawi of Ethiopia, who has strong record of repressing minority groups and political opponents.

Democratic accountability is important to governance, and fundamental in promoting development. To begin with the obvious Civics 101 view, accountability is a crucial mechanism to ensure that government does good and not ill to those affected by its actions. Under democracy, citizens can use many mechanisms -- such as voting, popular protests, and spoken and written criticisms -- to penalize governments that are harming individuals (even if it is only a minority of individuals).

The same mechanisms reward political actors that do good by, for example, supplying public goods. When such mechanisms work, the government is accountable to its citizens. The opposite of accountability is impunity -- the government can do whatever it wants to its citizens without consequences.

>>Download the paper