May 2011
Sun Mon Tue Wed Thu Fri Sat
1 2 3 4 5 6 7
8 9 10 11 12 13 14
15 16 17 18 19 20 21
22 23 24 25 26 27 28
29 30 31        

Authors' Committee

Chair:

Matt Blackwell (Gov)

Members:

Martin Andersen (HealthPol)
Kevin Bartz (Stats)
Deirdre Bloome (Social Policy)
John Graves (HealthPol)
Rich Nielsen (Gov)
Maya Sen (Gov)
Gary King (Gov)

Weekly Research Workshop Sponsors

Alberto Abadie, Lee Fleming, Adam Glynn, Guido Imbens, Gary King, Arthur Spirling, Jamie Robins, Don Rubin, Chris Winship

Weekly Workshop Schedule

Recent Comments

Recent Entries

Categories

Blogroll

SMR Blog
Brad DeLong
Cognitive Daily
Complexity & Social Networks
Developing Intelligence
EconLog
The Education Wonks
Empirical Legal Studies
Free Exchange
Freakonomics
Health Care Economist
Junk Charts
Language Log
Law & Econ Prof Blog
Machine Learning (Theory)
Marginal Revolution
Mixing Memory
Mystery Pollster
New Economist
Political Arithmetik
Political Science Methods
Pure Pedantry
Science & Law Blog
Simon Jackman
Social Science++
Statistical modeling, causal inference, and social science

Archives

Notification

Powered by
Movable Type 4.24-en


« May 2, 2011 | Main | May 25, 2011 »

19 May 2011

Pre-registration for Studies?

In response to a comment by Chris Blattman, the Givewell blog has a nice post with "customer feedback" for the social sciences. Number one on the wish-list is pre-registration of studies to fight publication bias -- something along the lines of the NIH registry for clinical trials.

I couldn't agree more. I especially like that Givewell's recommendations go beyond the usual call for RCT registration to suggest that we should also be registering observational studies. If we're dreaming about discipline-wide reforms to enhance the credibility of political science, it would be nice if we had reforms that weren't only applicable to the research that is already most credible.

The most though-provoking reform idea thrown out by the Givewell blog is this:

As food for thought, imagine a journal that accepted only studies for which results were not yet known. Arguably this journal would be more credible as a source of "well-designed studies addressing worthwhile questions, regardless of their results" as opposed to "studies whose results make the journal editors happy."
[Thought experiment round two: how would this journal differ from the APSR?]

I've been trying to think of ways to personally implement the principle of preregistration (short of organizing a registry or starting the above journal). The most obvious thing I can think of is to keep a detailed lab notebook (see discussion by Lupia here). Ideally, it would be public so that I couldn't go back and fudge it -- "Oh, I expected all along that the coefficient would be negative." Or maybe I'd keep it private during the research but somehow make deletions impossible.

Actually, even if I never made this public, taking better notes as I do research could have serious benefits. For one thing, it would be incredibly helpful for mitigating the inevitable bit-rot from letting a project sit for a while. And it's nice to be able to remember how a project actually unfolded. As Fox sagely observes, "It is best...not to fool yourself, regardless of what you think about fooling others" (p. 511, in reference to standard errors).

Perhaps there's actually a market for this kind of thing. Would reviewers look more favorably on papers submitted with a time-stamped preregistration? I guess not, or else at least a few people would be doing it already.

Still, I'm tempted to give public lab notes a whirl myself. Suggestions welcome!

Posted by Richard Nielsen at 7:58 PM