April 2009
Sun Mon Tue Wed Thu Fri Sat
      1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30    

Authors' Committee

Chair:

Matt Blackwell (Gov)

Members:

Martin Andersen (HealthPol)
Kevin Bartz (Stats)
Deirdre Bloome (Social Policy)
John Graves (HealthPol)
Rich Nielsen (Gov)
Maya Sen (Gov)
Gary King (Gov)

Weekly Research Workshop Sponsors

Alberto Abadie, Lee Fleming, Adam Glynn, Guido Imbens, Gary King, Arthur Spirling, Jamie Robins, Don Rubin, Chris Winship

Weekly Workshop Schedule

Recent Comments

Recent Entries

Categories

Blogroll

SMR Blog
Brad DeLong
Cognitive Daily
Complexity & Social Networks
Developing Intelligence
EconLog
The Education Wonks
Empirical Legal Studies
Free Exchange
Freakonomics
Health Care Economist
Junk Charts
Language Log
Law & Econ Prof Blog
Machine Learning (Theory)
Marginal Revolution
Mixing Memory
Mystery Pollster
New Economist
Political Arithmetik
Political Science Methods
Pure Pedantry
Science & Law Blog
Simon Jackman
Social Science++
Statistical modeling, causal inference, and social science

Archives

Notification

Powered by
Movable Type 4.24-en


« April 7, 2009 | Main | April 12, 2009 »

9 April 2009

Is randomization really a panacea?

In the spirit of Weihua's post on inference in non-random and random settings, I found this paper by Anup Malani on selection bias in randomized studies. The thirty second point is that patients' prior beliefs over outcomes of a study matter and as the probability of being in the "good" arm (to the extent that there is one) increases, the marginal patient will expect a smaller effect from the study. What this means is that the study population is not representative of the overall population and that real-world outcomes will always be worse than those observed in an experiment since in the real world the probability of receiving the "good" treatment is 1.

Malani's model proposes three potential treatments--none, conventional, and experimental--and each treatment has a heterogeneous effect across the population; for simplicity I will assume that the outcome is binary (alive/dead, employed/unemployed, elected/unelected, etc.). Individuals have beliefs about the probability of a good outcome for each treatment and there is also a distribution of true probabilities of a good outcome for each treatment. The key assumption is that these two distributions are not independent, thus knowing an individual's preferences over the three potential treatments provides information about true outcomes (as long as it doesn't provide too much information). The exciting part of this paper is that when one thinks of an experiment, it is simply a lottery among these alternatives, perhaps a lottery between no treatment and the experimental treatment. Then anyone who believes no treatment is better than conventional treatment should enroll in the study, regardless of the lottery. The interesting case is when should individuals who believe that conventional treatment is better than no treatment take the risk of entering the study and potentially being assigned to no treatment? Intuitively this happens when an individual believes that the expected benefit of entering the study is greater than taking conventional treatment, but this benefit depends on the lottery. Therefore, in a simplified form of Malani's argument, as the probability of assignment to the new treatment increases I will enroll in the study with a weaker belief in the new treatment. Malani then shows that under plausible assumptions this yields selection bias and demonstrates that this bias could explain much of the benefit observed in randomized studies of anti-ulcer drugs.

Thus even randomized studies are not immune to selection bias.

What does one do to address this problem? Malani proposes allowing subject to choose which control they would like to receive--thus when a subject is enrolled, she can choose a lottery over no treatment and new treatment or over conventional treatment and new treatment. This then provides estimates of the treatment effect for individuals who prefer conventional treatment to no treatment, which may be the most relevant for policy purposes.

Posted by Martin Andersen at 2:00 PM