How to Evaluate Market Research Reports

Research Validity and Reliability

Whether research reports are sold for a fee to help marketers make decisions or are given out free to pump up a company’s products, the significance of the information contained in the report will depend on how well the research was conducted.  But doing research right takes a good deal of knowledge of research methods and a lot of time.  By “right” we are talking about using scientific methods that have been tested and refined over hundreds of years and hold up to statistical analysis.  Unfortunately, many research reports, especially those produced for promotional reasons, are not scientific and thus may not carry much value.

One of the biggest concerns when reading a research report is whether what is presented is indeed relevant and worth considering.  To be relevant the research must be designed in a way that overcomes a number of obstacles particularly in terms of research validity and research reliability. 

• Research Validity - This represents several concepts that to the non-researcher may be quite complex.  But basically validity boils down to whether the research is really measuring what it claims to be measuring (see Purpose of Research below).  For instance, if a company says its report is measuring how people prefer its products over competitors’ products, is the research really designed in a way that tests this?  While research validity is measured in several ways, most of which are beyond the scope of this tutorial, the simple concept of “is it researching what it is supposed to be researching” should be a question that is at the top of the list when evaluating a research report.
• Research Reliability - This is measured in terms of whether research results can be applied to a wider group than those who took part in the study.  In other words, would similar results be obtained if another group containing different respondents were used?  For example, if a CPRR shows results of a few focus groups with a total of 40 people participating, is the information obtained from these people sufficient to conclude how the entire target market for a product, which may number in the millions, feels about the company’s products?  Or is it likely that if 40 different people took part in another set of focus groups then their responses would be different?

As we will see, most of the issues we will discuss impact research validity and reliability.  If concerns arise for one or both of these issues, then the quality of the research design and its results are suspect.