20 January 2006
The past two days have seen Jon Krosnick present his new book, The Handbook of Questionnaire Design: Insights from Social and Cognitive Psychology to a packed conference at Harvard. Krosnick's aim is an extraordinary one - to distil the thousands of studies that look at how individuals respond to survey questions and derive basic rules about how to design reliable and valid questions.
Much of the past two days has focused on the "mechanics" of survey questions - how many response options should we give respondents and how should we order them; should we use rating scales or ranking methods; should we offer a "don't know" response and so on. One of today's sessions stood out, however, for its lack of precise rules.
Jon discussed how to word questions - and basically offered two pieces of guidance. First, use clear, unambiguous language to ask directly about the item of interest. This seems straightforward to me. This fall, I wrote some questions for a poll with Steve Ansolabehere at MIT, and largely relied on my own instincts and the advice of colleagues to make sure the questions were unproblematic.
But Krosnick's second piece of advice was basically not to trust your own instincts, since survey respondents often interpret questions in ways that are not predictable when sitting in a university office! His answer to the problem is to pre-test questions, asking respondents to explain what they understood the question to be asking. The poll I worked on - and probably any poll that I will work on whilst a grad student - didn't have the resources to run this kind of pre-test. Presumably other researchers also have this problem: does anyone have a good solution to cheaply and conveniently check how ordinary respondents will react to the questions we ask?
Posted by Phil Jones at January 20, 2006 8:31 PM
I believe that some survey firms (e.g., Knowledge Networks) include the price of a pretest in their total survey costs--and this should be part of the discussion when you are selecting a survey firm (as well as part of the budget in your grant proposals if necessary). But even if you cannot pretest on a random sample, you should pretest the questions on family, friends, students, neighbors, etc who are outside your field of study. Test it out on your grandmother--you get brownie points for calling her and she is likely to give you exactly the feedback that you need. I think the importance of pretesting cannot be emphasized enough. I believe that some large surveys have the policy that they will only use questions that have been previously fielded. Otherwise, you risk wasting a great deal of money on a useless question.
Posted by: Sunshine Hillygus at January 21, 2006 2:39 PM