2 February 2010
The other week, I read Jared Diamond's Guns, Germs, and Steel which managed to get me a little worked up about a pet peeve of mine: the term "natural experiment." Just when I had gotten calmed down, the Polmeth list serve alerted me to an entire issue of Political Analysis devoted to natural experiments. Arghhh...
Don't get me wrong -- in my own research I try to use observational data to make causal claims that are probably far more dubious than anything in the special issue of Political Analysis. I'm highly impressed by the research and I'm even more supportive of social scientists who are looking for "natural experiments" in political science. I just wish we could call them something else because I'm skeptical that they are really experimental.
The lead article of the PA special issue urges scholars "to use the language of experimental design in explicating their own research designs and in evaluating those of other scholars." I'm on board with using the language of experiments, but I've also seen more than a few recent papers framed as "natural experiments" that are really just observational studies with no particular claim to special status. The spread of experimental language into observational studies may have downsides as well as benefits.
Until recently, I basically assumed that when people said they had a natural experiment, what they really meant was that they had a credible instrument: a variable that breaks the link between treatment assignment and the potential outcomes for some or all of the units. However, the lead PA article places difference-in-differences, regression discontinuity, and matching methods under the tent of natural experiments. While I like (and use) these techniques and find them compelling, only some of them explicitly rely on an IV-type argument. Maybe I have more to learn.
The problem with any randomization that isn't controlled by the researcher is that extreme skeptics like me can then try to spin complicated stories about how confounding could occur. This is what I found myself doing while reading Guns, Germs, and Steel. An extremely simplified version of Diamond's argument is that geography, not genetics, determines which human societies become dominant and which are conquered or destroyed. He devotes the entirety of chapter 2 to discussing the settlement of Polynesia by people who come from essentially the same genetic stock but experienced different geographies once they settled particular islands. The random variation in geography is interpreted as the cause for significant variation in the trajectories of the peoples of each island or group of islands.
This might be a natural experiment if Diamond could show that people were somehow randomly assigned to different islands. The problem is that different types of people might chose to live on different islands. Although it may be random which islands an exploratory party reaches, the explorers can choose to stay or move on for reasons that might be related to genetic variation. Similarly, explorers and colonists are probably not a random sample of the population, so the types of people that reach a far off island might have different genetic traits than those that remain in already established population centers. You get the idea.
I should reiterate that these reservations are just my gut reactions rather than a well thought-out assault on the use of natural experiments. I'm interested to read more: Jared Diamond and our very own James Robinson have a new book out on the subject that I'm excited to read. Thad Dunning has written on the topic, as have others.
Bottom line: I'm thrilled (and jealous) whenever social scientists find some plausibly exogenous variation to exploit for causal inference. I think it should happen more. I just worry that by attaching the "experimental" label to these studies, we endow them with undue credibility.
Posted by Richard Nielsen at February 2, 2010 11:30 AM