Very early in my market research career, I was fortunate to work in a quality “bubble”. It was 2003 and I began filling market research project quotas as a Project Manager working with a start-up called e-Rewards. In my experience back then, sample quality was not an issue since the panel was sourced “by invitation only” ensuring real people enrolled from reputable corporate programs. I had fielded at least 100 studies before the topic of sample quality was raised in the slightest.
I clearly remember the day when my naive quality bubble burst. A client contacted us to express concern about identical responses to an open-ended question from dozens of “unique'' respondents. After a brief investigation, I was shocked to learn a panelist had figured out a way to game our sampling system in a failed attempt to earn multiple survey incentives. My sampling honeymoon was over. The need to separate “accepted” vs. “rejected” completes (aka reconciliation) became a standard process that continues to this day.
As the industry evolved over time, it became clear that the sample quality problem was increasing in magnitude. Survey “gaming” seemed to shift from an individual novelty, to commonplace, to illegitimate small businesses effectively leveraging technology to operate at scale.
In parallel, demand for online sample significantly outpaced online sample supply. Traditional panels struggled to keep up. Single source sample plans that were once commonplace became less reliable for medium to large studies. Aggregating sample from multiple sources increased in frequency to achieve quota targets previously feasible from one source. Innovative new technologies such as sample routers and sample exchanges provided a sorely needed boost in supply...yet demand continues to exceed supply.
The COVID pandemic magnified our industry’s sample quality and supply problems. Sample suppliers were understandably reluctant to invest in maintaining pre-COVID supply levels. Supply declined significantly in many cases. The need to maximize return-on-supply acquisition costs was more important than ever. Sample demand dropped for several months but by most accounts, recovered much faster than anticipated. The research community seized an incredible opportunity to create the valuable insight companies needed to effectively adjust in their new business environment.
While the world around us has changed significantly, many surveys have not. Challenges such as poorly written content, disconnects between survey length and incentives, inadequate support for mobile devices, broken redirects, repeating demographic questions, and a variety of other factors often result in a poor respondent experience. The poor respondent experience results in low cooperation rates and data that needs to be “scrubbed” to remove fraudulent responses.
Suffice to say, if we hope to solve our sample quality and feasibility challenge, improvements are needed from both sample buyers and suppliers. Quality sample data collection has become a pandemic in itself. Many researchers and suppliers have worked to improve quality through effective research design, field processes, and innovative use of technology. Despite their best efforts, the problem remains unsolved. After nearly two decades, two truths appear to be evident. Broadly speaking…
- Sample suppliers are either unable or unwilling to consistently deliver quality sample
- Sample buyers are either unable or unwilling to consistently collect quality data
These statements are not intended to be harsh or judgmental. The factors that led to the current state are highly complex with significant reputational and financial ramifications. Rationale aside, “it is what it is” as we say in Texas.
If history is any guide, large-scale change will not happen without large-scale intervention. To illustrate this point, let’s review market research industry content over the past 18 years that shows a persistent problem:
- 2003, ARF – The Research Authority: “Internet surveys are the newest research data collection format...Longer surveys are possible with good incentives although data quality may be jeopardized. Internet surveys are relatively inexpensive.”
- 2007, Baker, R. & Dowens: “Separating the wheat from the chaff: ensuring data quality in Internet samples”
- 2008, MRWeb: “Our research demonstrates that problems in sample quality have a direct impact on research results and can dramatically increase the risk that poor data quality will translate into bad business decisions.”
- 2009, Forrester: “Since the beginning of online surveys, there have been questions about how clean the online panels that enable them are. Questions abounded about representativeness, fraud, professional survey takers, inattentive survey takers and the like.”
- 2010, AAPOR Report: “...industry trade associations, professional organizations, panel companies, and individual researchers have all focused on the data quality issue.”
- 2014, Insights Association: “It’s estimated that 25 percent of online data collected is of poor quality due to the respondents’ lack of authentic interest in the survey. This has made data quality an ongoing topic of debate among researchers.”
- 2015, Greenbook: “Heal Thyself! When Will Market Research Get Serious About Sample Quality?”
- 2016, Journal of Business Research: “With the exploding use of Internet surveys, research efforts and data quality are increasingly subject to the effects of respondents who do not give the required attention to survey questions and who speed through the survey, or who intentionally cheat with their answers.”
- 2018, Greenbook GRIT: “Issues surrounding accessing quality and representative sample is without doubt the single biggest individual challenge mentioned in the survey.”
- 2020, PEW Research: “Assessing the Risks to Online Polls from Bogus Respondents”
The famed motivational speaker Zig Ziglar comes to mind. He once said, “They’re in denial. They think denial is a river in Egypt!” In this case, “they” represents our market research industry. The bottom line: the market research ecosystem needs to evolve to ensure the long-term success of our industry.
Given the long stalemate between sample suppliers and buyers, an improved partner ecosystem is needed to bridge the gap. Suppliers and buyers alone cannot solve our sample quality and feasibility challenges.
One way the industry is evolving is through the introduction of “Market Makers”. In his recent blog entitled “ResTech and Beyond: The Continued Expansion of Our Industry Landscape”, Lucid CEO, Patrick Comer, defined Market Makers as:
“…typically data collection and sample fielding specialists that provide buyers with the tools and services needed to succeed in their research objectives. These tools and services include matching buyers with sellers, programming and hosting, translation, data cleaning, data reporting, and more. Market Makers play an integral role in the overall ecosystem with a strong correlation to their role in financial services.”
A recent white paper from Deloitte about evolving partner ecosystems states “...driven by a shift in customer expectations from product delivery to the enablement of outcomes, companies are increasingly realizing the need to redefine their offerings, accelerate and broaden innovation, and deliver solutions in partnership with ecosystem partners.” In the world of market research specifically, Market Makers must add value by:
- Democratizing sample aggregation knowledge and best practices
- Partnering with researchers to combine their research expertise with practical research fieldwork best practices
- Improving sample quality through rigorous methodologies validated by unbiased third-party sources
- Streamlining the sample procurement and fulfillment process
A Market Maker can act as a research agent, providing unbiased consultation between sample buyers and suppliers. Sample source autonomy and impartiality is key for Market Makers to be effective. Their focus is finding win-win research solutions that are not biased by individual buyers or supplier goals. They can intelligently source multiple suppliers systematically to easily place and fulfill sample orders. They consistently apply rigorous quality assurance methodologies so researchers can be confident the insight they create comes from accurate data. Market Makers need a combination of talented, experienced people, well-designed business processes, and innovative technologies to add value at scale.
A recent article by McKinsey said it well: “Digital supply-chain transformation is about much more than technology. For the latest wave of supply-chain innovation to deliver its full potential, companies must be willing to adapt their processes, capabilities, and management systems. They need the willingness and flexibility to learn, adapt and change as they go. And above all, they need to ensure that their people are with them on the journey.”
Market Maker results can have a profound impact on industry for both buyers and suppliers. Buyers benefit from increased feasibility, increased quality, reduced field time, and lower cost. Suppliers benefit from lower operating expenses and higher returns on sample acquisition expenses, mostly in the form of higher traffic conversion rates, reduced incentives paid to fraudulent respondents, and lower sample reconciliation rates.
Private equity firm LLR Partners said, “We predict that in the next five years, traditional research technology incumbents will represent less than 50% of the total market, creating a massive opportunity for smaller, more tech-forward players.” This is resoundingly true for the sample space, as data collection firms and sample suppliers must stay ahead of challenging marketplace trends like fewer respondents, heightened demand for more respondents, niche audience requirements and, of course, the ever-present “need for speed.” Nimble, expert, tech-forward approaches that keep the Market Makers model in mind can deliver success for the future of sampling and the insights community.