How to Evaluate Market Research Reports

5. What method was used to obtain the responses?

While the research instrument may be well designed and respondents well chosen, a research report can still fail if the method for collecting the information is faulty.  Survey data can be collected in several ways including by mail, over the phone, online or in-person.   In an ideal research circumstance, researchers have full control over study participants’ environment.  For instance, a high level of control can exist when respondents complete surveys in research facilities managed by the researchers, such as in survey centers found at some shopping malls.  But this level of control is clearly not practical for other collections methods, especially when a large number of respondents is sought. 

In situations where respondents’ environment cannot be controlled, the results must be more closely examined.  Experienced researchers have tools to institute a good degree of control even when the respondents are not physically in their presence but, just like testing questions and generating good samples, instituting controls can be expensive and is often not pursued by researchers who lack resources or have little experience.

In addition to the respondent’s environment, some concerns may be raised if a respondent’s motivation for participating in research is tainted in some way.  Respondents should be actively screened to insure they not only meet the sample requirements but that their motives for participating will not compromise their responses.  As we noted in our article Issues with Paying Respondents to Take Part in Online Research, researchers must carefully screen those who take part just to get an incentive and have little interest in offering honest responses from those who want the incentive but are willing to share their views.  Unfortunately, while most research reports indicate how respondents were located, most do not go into detail on how respondents are screened.  Follow-up with the researcher may be necessary to determine this.