 |
A Case Study of “Over 3,500 Online Surveys!”
Online survey research is among the most rapidly growing segments of the multi-billion dollar marketing research business and one which plays a crucial role in guiding business decisions in virtually every industry. Indeed, a vast array of organizations now use the Internet to conduct research of current and potential customers in an effort to make profitable product development and marketing decisions.
Yet there are still many questions that have not been answered about the nature of respondent behavior in online studies and whether companies who purchase these studies typically get accurate data from authentic, qualified respondents. Put simply, the industry of marketing research seems to lack a respondent-centric theory of study behavior that will assist researchers and managers in our effort to effectively identify error and provide clients with the highest quality data that can be gathered.
Common approaches to identifying survey error include algorithms and other quantitative methods that are designed to locate patterns which analysts have identified as common among those who lie or provide “lazy” responses in surveys. But let us not forget that while mathematics is an invaluable tool to those who study consumer behavior, marketing research is really about people and much error seems to find its way into the PowerPoint presentations that are delivered to marketers and senior managers every day. Therefore, we feel that a qualitative approach to spotting error in online surveys is critical. In keeping with this philosophy, I have developed a brief case study which highlights some of the problems we’re uncovering in our ongoing study of frequent survey-takers.
The Case of “Mike” and his 3,500 Surveys
While we know that there are indeed honest respondents who tell the truth in surveys, we also know that there are many questionable respondents who most analysts would not want in one of their studies. These are the folks that sometimes lie to get an incentive or rush through a survey to make money, unconcerned about the integrity of data they offer.
Consider the case of “Mike”: Mike is a “professional survey-taker” who claims to have taken over 3,500 surveys. Bear in mind that Mike’s website links to some of the largest panels in the online survey industry (thesurveypro.com/Freesurveys) and that he is not only in the business of taking surveys but he also recruits respondents for large panels. Mike teaches would-be respondents how to “maximize” their survey-taking income using wine (he prefers Merlot) and a program called “RoboForm” that allows one to automate the process of filling out online forms. Let’s take a closer look at “Mike.”
In his website, Mike shares his experience taking online surveys and provides tips on how to maximize one’s survey-taking income (mikessurveys.org). So, how does Mike take so many surveys? Here are a couple of tips that Mike offers his visitors: Drink wine while taking surveys and use RoboForm to speed up the registration process.
"I usually do mine with a glass of wine..."
At the top of his “getting started” page designed for folks who are considering becoming “professional survey-takers” (mikessurveys.org/gettingstarted.htm), Mike says “[t]aking surveys should be fun. I usually do mine with a glass of wine in the evenings, but I cannot guarantee that will help you earn more money. :)” Thus, Mike gets drunk as he takes surveys so as to increase the “fun-factor.” Mike is clearly not the guy marketing analysts mention when trying to convince potential clients that they should use online panels to conduct their marketing research. Yet, my work indicates that people like Mike do indeed take surveys which form the foundation of many marketing and product development decisions.
After reviewing Mike’s website—and his is not the only one of its kind— it is indeed clear that his approach does not emphasize honesty, transparency, careful response consideration and depth. Rather, his goal is to make as much money as possible— as quickly as possible— taking online surveys.
RoboForm: How People Automate the Process of Taking Surveys
While Mike may be drunk when taking surveys, he is also rather tech. savvy and very focused on making money (as opposed to carefully considering his answers and telling the truth). Mike explains that one tool of his trade is “RoboForm,” a program designed to automate the process of filling out online panel registration forms. This program is promoted on several sites as a means by which respondents can speed up the process of registering with companies that pay people to take surveys. I wonder whether this is one of the programs that are used by those folks we have seen in datasets who complete a survey in less than two minutes when the average time-to-completion is 8-10 minutes.
The Qualification Scam
One of the most blatant examples of deception techniques we have found thus far came from another site (earnsurveymoney.com) that teaches would-be respondents “Tricks of the Qualification Survey.” While the tricks described in this site (earnsurveymoney.com/qualificationsurvey.html) are carefully worded so as not to invite liability, it’s clear that the site is teaching people how trick survey researchers in an effort to get an incentive. After all, these folks are not taking surveys because they simply want to help us develop great products or optimize marketing efforts; rather they are trying to make money. Unfortunately, my work indicates that some respondents do indeed use these “tricks” as a way to scam the qualification process.
Toward a Solution: Be Diligent and Consult with Experienced Analysts
To be sure, “Mike” the survey-taker doesn’t make money by emphasizing honesty, he makes money by taking as many surveys as possible—as quickly as possible—and recruiting others to do the same.
There are methods that can be used to detect deception as well as survey takers like Mike who are clearly in too much of a hurry— or simply too drunk— to carefully consider their responses and provide in-depth and honest feedback.
Contact Jerry Savage if you would like to learn more about how to detect respondent error in online surveys.
Yet there are still many questions that have not been answered about the nature of respondent behavior in online studies and whether companies who purchase these studies typically get accurate data from authentic, qualified respondents. Put simply, the industry of marketing research seems to lack a respondent-centric theory of study behavior that will assist researchers and managers in our effort to effectively identify error and provide clients with the highest quality data that can be gathered.
Common approaches to identifying survey error include algorithms and other quantitative methods that are designed to locate patterns which analysts have identified as common among those who lie or provide “lazy” responses in surveys. But let us not forget that while mathematics is an invaluable tool to those who study consumer behavior, marketing research is really about people and much error seems to find its way into the PowerPoint presentations that are delivered to marketers and senior managers every day. Therefore, we feel that a qualitative approach to spotting error in online surveys is critical. In keeping with this philosophy, I have developed a brief case study which highlights some of the problems we’re uncovering in our ongoing study of frequent survey-takers.
The Case of “Mike” and his 3,500 Surveys
While we know that there are indeed honest respondents who tell the truth in surveys, we also know that there are many questionable respondents who most analysts would not want in one of their studies. These are the folks that sometimes lie to get an incentive or rush through a survey to make money, unconcerned about the integrity of data they offer.
Consider the case of “Mike”: Mike is a “professional survey-taker” who claims to have taken over 3,500 surveys. Bear in mind that Mike’s website links to some of the largest panels in the online survey industry (thesurveypro.com/Freesurveys) and that he is not only in the business of taking surveys but he also recruits respondents for large panels. Mike teaches would-be respondents how to “maximize” their survey-taking income using wine (he prefers Merlot) and a program called “RoboForm” that allows one to automate the process of filling out online forms. Let’s take a closer look at “Mike.”
In his website, Mike shares his experience taking online surveys and provides tips on how to maximize one’s survey-taking income (mikessurveys.org). So, how does Mike take so many surveys? Here are a couple of tips that Mike offers his visitors: Drink wine while taking surveys and use RoboForm to speed up the registration process.
"I usually do mine with a glass of wine..."
At the top of his “getting started” page designed for folks who are considering becoming “professional survey-takers” (mikessurveys.org/gettingstarted.htm), Mike says “[t]aking surveys should be fun. I usually do mine with a glass of wine in the evenings, but I cannot guarantee that will help you earn more money. :)” Thus, Mike gets drunk as he takes surveys so as to increase the “fun-factor.” Mike is clearly not the guy marketing analysts mention when trying to convince potential clients that they should use online panels to conduct their marketing research. Yet, my work indicates that people like Mike do indeed take surveys which form the foundation of many marketing and product development decisions.
After reviewing Mike’s website—and his is not the only one of its kind— it is indeed clear that his approach does not emphasize honesty, transparency, careful response consideration and depth. Rather, his goal is to make as much money as possible— as quickly as possible— taking online surveys.
RoboForm: How People Automate the Process of Taking Surveys
While Mike may be drunk when taking surveys, he is also rather tech. savvy and very focused on making money (as opposed to carefully considering his answers and telling the truth). Mike explains that one tool of his trade is “RoboForm,” a program designed to automate the process of filling out online panel registration forms. This program is promoted on several sites as a means by which respondents can speed up the process of registering with companies that pay people to take surveys. I wonder whether this is one of the programs that are used by those folks we have seen in datasets who complete a survey in less than two minutes when the average time-to-completion is 8-10 minutes.
The Qualification Scam
One of the most blatant examples of deception techniques we have found thus far came from another site (earnsurveymoney.com) that teaches would-be respondents “Tricks of the Qualification Survey.” While the tricks described in this site (earnsurveymoney.com/qualificationsurvey.html) are carefully worded so as not to invite liability, it’s clear that the site is teaching people how trick survey researchers in an effort to get an incentive. After all, these folks are not taking surveys because they simply want to help us develop great products or optimize marketing efforts; rather they are trying to make money. Unfortunately, my work indicates that some respondents do indeed use these “tricks” as a way to scam the qualification process.
Toward a Solution: Be Diligent and Consult with Experienced Analysts
To be sure, “Mike” the survey-taker doesn’t make money by emphasizing honesty, he makes money by taking as many surveys as possible—as quickly as possible—and recruiting others to do the same.
There are methods that can be used to detect deception as well as survey takers like Mike who are clearly in too much of a hurry— or simply too drunk— to carefully consider their responses and provide in-depth and honest feedback.
Contact Jerry Savage if you would like to learn more about how to detect respondent error in online surveys.