written by Jackie Lorch
It was Einstein who is supposed to have said “Everything should be made as simple as possible; but not simpler.” Sometimes the “obvious” reason for a problem with our research data is not the right explanation. Often we ascribe unusual research results to a change in the market, or to the sample source, or to the way the questionnaire was written. And just as often, we have to dig much deeper to understand what’s really going on.
Perhaps the results are attributable to the way we’re screening within the questionnaire, or we find that in our efforts to create a “NatRep” demographic sample, when “NatRep” isn’t the right choice for a particular project, we’ve introduced an unintended bias.
Or, there is a cultural misunderstanding when a question has originated in one country and been extended to others. One simple example of this is when an income question is asked across countries. From a US perspective, income questions mean annual income. In many countries, such as Mexico, Brazil, Russia and China, people think about their income in monthly terms. When presented with a list of annual incomes in these countries, people will still respond with their monthly income and the resulting data will suggest extremely low incomes in these countries compared to others.
When the pace of business moves so fast and clients are pressuring us for answers, it’s tempting to settle on an obvious answer, like fraud or satisficing on the part of the respondent to explain data anomalies.
Apparently even Occam’s razor itself has suffered from over-simplification: Reportedly the 14th Century monk William of Ockham didn’t exactly say that all things being equal, a simpler explanation is better. Instead he said that we should move towards simpler explanations until simplicity can be traded for more explanatory power. A good principle for investigating research data questions.
Categories: Market Research