14 March 2006
"I'm doing a survey. I've never done this before, taken any classes on survey research, or read any books on the subject, and a friend suggested that I get some advice. Can you help me? I'm going in the field next week."
Someone has asked me versions of this question almost every month since I was a graduate student, and every time I have to convey the bad news: doing survey research right is extremely difficult. The reason the question keeps coming up is that it seems like a such a reasonable question: what could be hard about asking questions and collecting some answers? What could someone do wrong that couldn't be fixed in a quick conversation? Don't we ask questions informally in casual conversation all the time? Why can't we merely write up some questions, get some quick advice from someone who has tried this before, and go do a survey?
Well, it may seem easy, but survey research requires considerable expertise, not any less than heart surgery or flying military aircraft. Survey research should not be done casually if you care about the results. Survey research seems easy because its possible to learn a little without much expertise, whereas doing a little heart surgery with a dinner knife, or grabbing the keys to a B-2 after seeing Top Gun, wouldn't accomplish anything useful.
Survey research is not easy; in fact, its a miracle it works at all. Think about it this way. When was the last time you had a misunderstanding with your spouse, a miscommunication with your parent or child, or your colleague thought you were saying one thing and you meant another? That's right: you've known these people for decades and your questions are still misunderstood. When was the last time your carefully worded, and extensively rewritten article or book was misunderstood? This happens all the time. And yet you think you can walk into someone's home you've never met, or do a cold call on the phone, and in five minutes elicit their inner thoughts without error? Its hard to imagine a more arrogant, unjustified assumption.
So what's a prospective survey researcher to do? Taking a course, reading some books, etc., would be a good start. Our blog has discussed some issues in survey research before, such as in this entry and this one on using anchoring vignette technology to deal with the problem of survey respondents who may interpret survey questions differently from each other and from the investigator. Issues of missing data arise commonly in survey research too. I'm sure we'll discuss lots of other survey-related issues on this blog in the future as well.
A more general facility for information on the subject is the Institute for Quantitative Social Science's Survey Research Program, run by Sunshine Hillygus. This web site has a considerable amount of information on the art and science of questioning people you don't know on topics they may know. If readers are aware of any resources not listed on this site that may be of help survey researchers, please post a comment!