Finally, the definitive guide to creatively manufacturing your own research result

From the brilliant xkcd (also the creator of this classic in statistics humor).

We couldn’t resist using this as a way to illustrate some of our early wonky posts complaining about the suspected practice of “data mining” in aid research.

In aid world, research looks for an association of some type between two factors, like economic growth and foreign aid. But since both growth and aid contain some random variation, there is always the possibility that an association appears by pure chance.

“p < .05” is our assurance from the researchers that the probability that their result came about by coincidence is less than 1 in 20, or 5 percent, which is the accepted standard.

But the aid researchers—like the jelly bean scientists—are eager to find a result, so they may run many different tests. The problem, as Bill explained it, is that:

The 1 in 20 safeguard only applies if you only did ONE regression. What if you did 20 regressions? Even if there is no relationship between growth and aid whatsoever, on average you will get one “significant result” out of 20 by design. Suppose you only report the one significant result and don’t mention the other 19 unsuccessful attempts.…In aid research, the aid variable has been tried, among other ways, as aid per capita, logarithm of aid per capita, aid/GDP, logarithm of aid/GDP, aid/GDP squared, [log(aid/GDP) - aid loan repayments], aid/GDP*[average of indexes of budget deficit/GDP, inflation, and free trade], aid/GDP squared *[average of indexes of budget deficit/GDP, inflation, and free trade], aid/GDP*[ quality of institutions], etc. Time periods have varied from averages over 24 years to 12 years to to 8 years to 4 years. The list of possible control variables is endless….So it’s not so hard to run many different aid and growth regressions and report only the one that is “significant.”

And the next thing you know, there’s a worldwide boycott of green jelly beans…

UPDATE by Bill 12 noon: I asked around some journalist contacts of Aid Watch at leading newspapers how much awareness of this problem there is in the media, and got a fairly clear answer of ZERO.