How to Evaluate Market Research Reports

4. Who were the respondents in the research?

One of the biggest issues with survey research relates to those who participate.  Most research reports examine results of responses from a small percentage (i.e., sample) of a bigger group (i.e., population).  For instance, a major online retailer may conduct research on customers’ feelings toward the retailer’s product offerings by drawing a sample from customers who have purchased within the last year, of which there may be hundreds of thousands. 

When done right, sampling can produce very good information that can then be extended to the full population.  But the restrictions for doing sampling correctly are very tight and include:

• Random Sample Selection – At its strictest level sampling must ensure that all cases (e.g., customers) within the desired population (e.g., all customers who purchased in the last year) have the same chance of being selected to participate.
• Right Respondents Selected – This condition relates to sampling accuracy and making sure the people who do participate are in fact the ones being targeted (e.g., actual customer and not customer’s spouse).
• Sufficient Number of Respondents (Sample Size) – Good research requires the use of statistical calculations to determine the sample size needed to produce useful results.  While the methods for determining sample size are beyond the scope of this tutorial, any report that does not indicate the number of survey respondents should be suspected of not meeting minimum sample requirements.
• Non-Respondents Are Not Different (Non-Response Bias) – Not everyone who is asked to participate in a survey will actually agree to do so.  In fact, most research studies report a response rate which indicates how many people responded in relation to how many were invited to respond.  To ensure that those who do not respond are not different than those who do respond (e.g., do not have different characteristics, such as age or education level, that could affect interpretation of results), researchers should make a follow-up effort to find out about those that did not originally respond.  However, undertaking non-respondent follow-up is time-consuming and expensive, consequently few researchers ever bother to carry out this important analysis.  Those that do and report it are generally considered solid research designers.

Quality research reports will clearly report their methodology for selecting a sample.  If such information is not presented then a red flag should be raised.