December 2006
Sun Mon Tue Wed Thu Fri Sat
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30

Authors' Committee


Matt Blackwell (Gov)


Martin Andersen (HealthPol)
Kevin Bartz (Stats)
Deirdre Bloome (Social Policy)
John Graves (HealthPol)
Rich Nielsen (Gov)
Maya Sen (Gov)
Gary King (Gov)

Weekly Research Workshop Sponsors

Alberto Abadie, Lee Fleming, Adam Glynn, Guido Imbens, Gary King, Arthur Spirling, Jamie Robins, Don Rubin, Chris Winship

Weekly Workshop Schedule

Recent Comments

Recent Entries



SMR Blog
Brad DeLong
Cognitive Daily
Complexity & Social Networks
Developing Intelligence
The Education Wonks
Empirical Legal Studies
Free Exchange
Health Care Economist
Junk Charts
Language Log
Law & Econ Prof Blog
Machine Learning (Theory)
Marginal Revolution
Mixing Memory
Mystery Pollster
New Economist
Political Arithmetik
Political Science Methods
Pure Pedantry
Science & Law Blog
Simon Jackman
Social Science++
Statistical modeling, causal inference, and social science



Powered by
Movable Type 4.24-en

« December 13, 2006 | Main | December 15, 2006 »

14 December 2006

Misunderstandings among Experimentalists and Observationalists

We had several discussions a while ago on this blog about balance test fallacies, and an early version of a paper on the subject that Kosuke Imai, Liz Stuart and I wrote. Kosuke, Liz, and I also had a number of interesting discussions with people in several other fields about this topic, and we've found much confusion about the benefits of the key portions of the major research designs. Observationalists seem to have experiment-envy, which is in at least some cases unwarrented, and experimentalists have their own related issues too. To sort these issues out (largely or at least at first for ourselves), we have now written a new paper that tries to clarify these issues and also incorporates the points from the previous paper (material from the previous paper is the last few pages of this one). We'd be very grateful for any comments anyone might have.

"Misunderstandings among Experimentalists and Observationalists: Balance Test Fallacies in Causal Inference" by Kosuke Imai, Gary King, and Elizabeth Stuart.


We attempt to clarify, and show how to avoid, several fallacies of causal inference in experimental and observational studies. These fallacies concern hypothesis tests for covariate balance between the treated and control groups, and the consequences of using randomization, blocking before randomization, and matching after treatment assignment to achieve balance. Applied researchers in a wide range of scientific disciplines seem to fall prey to one or more of these fallacies. To clarify these points, we derive a new three-part decomposition of the potential estimation errors in making causal inferences. We then show how this decomposition can help scholars from different experimental and observational research traditions better understand each other's inferential problems and attempted solutions. We illustrate with a discussion of the misleading conclusions researchers produce when using hypothesis tests to check for balance in experiments and observational studies.

Posted by Gary King at 9:26 AM