There has been a good deal of talk in the news – and the marketing research industry – lately concerning the problems with modern day polling. One of the articles we discussed contained several quotes.
I think there's space in the market for a half-dozen kind of polling analysts.Nate Silver
What’s in a Problem?
Like many companies, the Marketing Workshop holds Lunch & Learns on a regular basis. Sometimes we bring in an outsider to present something of interest, but often times it is strictly an internal gathering. It’s important for us to do this because there are too few times that we are all able to get in one room together. Last week was one such occasion. With the election season getting ready to enter into full swing, one of our topics was polling. There has been a good deal of talk in the news – and the marketing research industry – lately concerning the problems with modern day polling. One of the articles we discussed contained several quotes that really got our discussion going.
“…it's getting hard to escape the conclusion that traditional ways of measuring public opinion no longer seem capable of accurately predicting outcomes the way they more or less once did.” “This overturns the philosophy behind probability polling, which holds that a relatively small number of people can stand in for the full population as long as they are chosen randomly from the larger group. If, on the other hand, the people who actually take the survey are systematically different from those who don't—if they're older, or have a different education level, or for some other reason are more likely to vote a certain way—the methodology breaks down.”
After reading these quotes, along with a few others, we asked ourselves several questions, “How do these things impact us as marketing researchers?” And, “Is our problem bigger or smaller than the polling industry?” I’m going to briefly share some of our thoughts and discussion, but mostly I want to throw the question out to the reader… What do you see as the similarities and differences of the problems that polling companies are facing with those we face in the Marketing Research industry?
At the onset of the discussion, it seemed that the general consensus was that the problems were near identical. Mainly focused on our ability (or lack thereof) to be able to reach a representative sample of the general population. As the debate went on however, and the thoughts started to bounce off each other and join together in different ways, the belief started to morph just a bit. (As an aside, I love how ideas behave like molecules and atoms when exposed to a collaborative and stimulating environment! I’ll save more on that for another post, though.)
Where we netted out (mostly) was that we are not in as bad a situation as the pollster. While the pollster is trying to predict a very discrete event – one with an ultimate “true” answer (who won the election?"), marketing researchers are trying to provide guidance about things like what might happen; who might buy, how much will they buy and so forth. The “answers” we need to come up with are much grayer in nature than the pollsters harsh black and white. For this reason, (and some other interesting ones) we decided that we are in a better position to deal with the methodological challenges then are the polling companies.
One other particular thought voiced and discussed was that what we are facing now is really not fundamentally different from when the industry moved from being primarily paper survey dominated to the telephone, and then from the telephone to being web-based. Do you agree? Is this really just the same discussion being debated over and over again?
I really am curious about others’ thoughts on these questions. Please comment with your take on all of this. And along the way, during this upcoming year where the news will be filled with stories of elections and polling, remember this particular quote (another we discussed), “Those paying close attention to the 2016 election should exercise caution as they read the polls.”
Looking forward to reading your thoughts and comments below…
~ Bud Sanders