In recent years, quantitative research has emphasized greater statistical power paired with advanced survey methodologies to drive insights and impress clients. Many new-age techniques such as sentiment analysis or digital ad tracking have begun to dominate online-based data collection. In other surveying, equally statistically rigorous yet more sophisticated methods such as Adaptive Choice-Based Conjoint (ACBC) or Total Unduplicated Reach and Frequency (TURF) analyses are becoming commonplace.
This strong stats strategy is the foundation of contemporary market research: sufficiently large n and sufficiently representative quotas can best deliver insights. But does our current system really best ensure high-quality data?
It is crucial to understand that perhaps the most important factor in collecting quality data is not the size of the sample, or even the methods used to survey the sample, but rather the quality of the sample in the first place. In their book Applied Survey Sampling, Edward and Johnny Blair Jr. explore the evolution of sampling methods over time, explaining not only that 1) “the who of sampling is more important than the how many,” but that 2) “a fair chance of selection to every member of the population” is essential for meaningful data collection.
This second factor – a fair chance of selection – is a major shortcoming of the current market research survey landscape. Too often, researchers simply rely on quota sampling to ensure that they recruit a group of respondents who are representative enough of their population of interest. However, this method is not without its flaws.
A serious problem with modern sampling procedures is that they often rely on Internet-based recruitment methods to secure their samples. Increasingly, more generalized sampling methods that could better capture a greater share of the members of a given population have been discarded for what amounts to online convenience sampling.
Sampling methods today rely on sample providers’ respondent lists and on biased recruitment methods through digital interfaces. Unfortunately, the distribution of high levels of Internet access is unbalanced within the population. Often sample providers fail to deliver respondent samples that are truly representative of the entire population – let alone of a particular consumer base or niche market.
Consider, for example, a CPG company that wishes to understand the appeal of one of its offerings within two major markets: The East and West Coasts. They figure that n respondents from each coast are sufficient for analysis. Additionally, they wish to understand how appeal varies across gender, and so request for the sample to be a 50-50 split of men and women.
Sometimes, sample providers or market research firms simply select respondents who fit any one of the desired quotas, without balancing the respondents across all conditions. Imagine, in this case, that all the desired men ended up coming from the West Coast and that all the desired women ended up from East Coast. Could we really provide any insights about the men or women, or East or West Coasts?
Unless nested quota sampling is used to ensure even distributions of women and men in each geographic location, the validity of the data would be compromised.
Furthermore, selections within each quota group are themselves subject to non-probability sampling. What if the sample provider disproportionately offers respondents to the research who are of a particular age or other demographic feature? What if the “population” we are drawing from isn’t really the true population?
Finally, what if all the facets of your population of interest are recruited in the appropriate proportions, but then due to attrition or systemic issues in the data collection process, some groups are disproportionately represented over others? Can this issue really be swept under the rug by weighting your data or through other statistical means?
Indeed, your n may not matter so much as your N. Sample size is of course important, but the quality of the sample is at least as important. Your survey research may have a sufficiently large sample, but if it is not representative enough of the true population, its validity must be called into question.
Often, to cut costs and save money, researchers use the first respondents they can find until they fill their required quotas. However, sparing the extra expense to reach out to more of the population can mean the difference between delivering real insights – and simply reporting statistics. Drawing on a larger proportion of the population and allowing for a much higher rate of respondent rejection is absolutely necessary for improving the quality of your market research data.
Ultimately, the onus falls on market research firms to raise industry standards, put the extra effort into their market research surveys, and have the hard conversations with clients about raising costs to secure stronger samples.
Of course, sampling issues stem not only from the number or variety of respondents recruited but from the quality of the data collected from them. Sometimes, respondents may not answer surveys well, or give up before they give you what you’d like. Sometimes, automated bots may even be completing surveys for them.
In another MDRG blog post written by Kristy Roldan, screening and data processing methods are explored that can limit the effects of fraudulent or low-quality data – which can creep up from even the most perfectly curated sample.
You can do everything right to secure good data for your population of interest: recruiting consistently and fairly for your sample, ensuring equivalent rates of response across quotas, screening participants, blocking bots from submitting fraudulent data, and all the rest. But does that mean that the data is high-quality?
Unfortunately, respondents often suffer from survey fatigue and struggle to deliver high-quality data across an entire questionnaire. What’s more, respondents are caught in a financial incentivization culture that encourages them to complete as many surveys as possible, as quickly as possible. In many cases, surveying is tedious and downright boring for respondents.
Perhaps the key to ensuring your sample provides you with high-quality responses is in making the experience meaningful and rewarding to them (and not just monetarily). Convincing your sample that their responses really matter, and that they will positively impact their own consumer experiences in the future, may be the most effective way to motivate your sample to give good quantitative data for your research project. What’s more, it may lend a greater share of your population the intrinsic motivation they need to participate in the first place.
Start your market research off right by recruiting a strong sample and following through with sound data collection methodology. Our experienced research analysts can help.