The Value of Online Research Panels: Issue of Research Quality

A category of online searching that draws a large number of user queries on Google and other search engines deals with how people can make “easy money.” Phrases such as make money online, how to make money and make money fast are searched hundreds of thousands of times a month. While the listed results are what would be expected, such as online articles suggesting how someone can make money, what also often appears are advertisements proclaiming how money can be made by participating in online research, such as answering surveys or participating in focus groups.

Many of these advertisements are for legitimate market research firms that use the Internet as a method for building research panels (a.k.a., consumer panels). Such panels consist of a large number of people who have agreed to take part in the research conducted by the panel’s operator. In most cases, the research is being done on behalf of a client, such as a marketing organization.

Research panels have been around for many years and, for just as long, research companies have offered incentives to those who agree to participate. Nowadays most incentives are in the form of gift cards, coupons, special discount offers and, sometimes, cash. In fact, some research sites suggest “easy money” can be made by those taking part in online research to the point of stating that someone can earn hundreds of dollars a month. The offer of incentives combined with the power of the Internet has led to a surge in the number of people willing to participate in research panels.

However, the growth of pay-to-participate research raises several concerns among marketers who are contemplating using research panels. One important concern is whether the aggressive promotional methods used to recruit participants will lead to a quality panel. That is, what percentage of a panel, who are being paid an incentive to participate, will actually have any interest in providing useful information? The notion of gaining information from someone only if they are paid may lead some marketers to question the validity of this research method especially if they feel participants are just giving answers to make money. This, of course, will make a marketer wonder if the participants are being honest with their answers or just providing information to complete survey or focus group questions.

While appearances may suggest incentive-based recruitment may have a negative influence on research panels, from a market research perspective this may not be the case. It all depends on the quality of the respondents in the panel. To ensure quality research, operators of legitimate panels will attempt to screen those who have requested to participate. The level of screening is key to whether the research panel is one of quality. The best screening seeks to confirm a person’s intentions and verify their profile by such methods as follow-up phone calls, direct email correspondence or require completion of a highly detailed questionnaire. But screening efforts are expensive leading less reputable panels (and there are many of these) to simply accept what a person enters in their profile when they register. Obviously, it would seem easy for someone whose primary purpose is to make money to scam panels that do little screening by simply entering bogus information when they register.

For panels doing little screening, it may be a stretch to consider what they do as contributing to producing quality market research since their base of respondents may be neither representative of the market nor offer truthful responses. It is safe to say these sites are almost never involved in highly scientific research and, consequently, the information they produce is not very useful. Instead, much of this research is used to create “eye-candy” statistics, charts and graphs that may appear on promotional materials, including in advertisements or posted on websites, but are not grounded in proper research techniques.

While the use of incentives may provide a useful method for research companies to deliver many to a panel, the number of participants, while crucial for operating a panel, is not the sign of a good panel. Instead, any research panel that bills itself as offering quality research must be scrutinized for the type of respondents that make up its panel. To be considered a useful research tool, a panel must be representative (i.e., share similar characteristics) of the market to be studied. Quality panels must also ensure that members are providing truthful information and not just answering questions or engaging in behavior just to get paid.

Thus, when it comes to research quality, marketing organizations looking to use a panel for research should not be as concerned with whether a member of a research panel was incentivized to participate but whether the panel is making efforts to effectively screen those who did agree to take part. Marketers should not use a panel that undertakes little screening if they intend to conduct good research.

In Part 2 of this topic, we will explore the ethical issues posed by online research panels.