The Pew Research Center actually does an ongoing poll on one of these:
- A new Pew Research Center survey of 2,048 adults finds that about two-thirds of the public (66%) believes there are “very strong” or “strong” conflicts between the rich and the poor.
Suppose you were asked to what extent you thought there were conflicts between the rich and the poor. I don't know how you or others might respond, but I would ask, "What do you mean by 'conflicts' among the rich and poor?" Fistfights? Arguments? General dislike of each other? Well, you'd think that Pew might offer a definition, but they don't. So what, exactly, did they ask their respondents? In its summaries, the Pew Centers says that the question they asked is simply "In America, how much conflict is there between poor people and rich people: very strong conflicts, strong conflicts, not very strong conflicts, there are not [sic] conflicts?” But that's not really true. If you dig far enough down into their report, you find that the full question, which was read to the respondent by the Pew interviewer, was this: "In all countries, there are differences or conflicts between different social groups. In your opinion, in AMERICA, how much conflict is there between poor people and rich people: ..." So the respondents are first told that there are differences or conflicts between different social groups. How does this affect the likelihood that respondents will answer that "there are not conflicts," or "what do you mean by 'conflicts'?" Note that even respondents who "understand" the question will likely have varying ideas of its meaning.
So we have a survey question of no obvious meaning asked in a very biasing manner.
And then we have the NY Times dutifully reporting the results without any discussion of what it is supposed to mean ("Survey Finds Rising Perception of Class Tension" By Sabrina Tavernise, January 11, 2012 ).
I don't know about you, but I have enough
meaningless stuff in my brain already.
When I worked for the Institute for Applied Social Science in Germany, I was shocked at the unscientific way survey questions were formulated. Basically, the researcher just posed questions of personal interest without any justification, definition or thought.
ReplyDeleteIt's not much more meaningful than when someone answers your "Howya doin'?" with "Fine".
I agree. I think researchers often can't stand to limit their survey to just the questions--maybe 2 or 3--they really need to ask, so they throw in additional questions that "seem interesting." It's a difficult impulse to resist. (Respect for the time one is asking of the respondents is a good perspective keep in mind.) I was proud of one questionnaire I wrote where I knew exactly how each question would fit into the report I would write--not that I thought I knew the results, but I knew why I asked each question. Doing that requires thinking through the research very thoroughly before writing the questionnaire.
DeleteOne other notable personal experience was when I simply could not think of a way to ask an important question (about "how much discovery" there had been in a court case case), without making it very complicated, and so never asked it.
There is a lot to be said about surveys; I expect to have some blog posts on the topic.