How to Evaluate Market Research Reports

7. How were results presented?

The result of a researcher’s work is presented in the form of the research report.  For most research conducted using surveys, the report contains explanatory text (i.e., narrative) along with visual representation of the results usually in the form of tables, charts or graphs.  But reports readers should be aware that a great way to hide poorly performed research is to mask the results within a fancy printed report, slide show or video presentation.  The most common methods for hiding poor research include:

• Report Only Positive Results – A sure sign that research is not what it is appears is when the report only presents a few results which often happens in reports where only issues that make the company look good are presented.  Unfortunately, the extent of the research problem may never be known unless the company offers access to the full survey instrument in order to see what was not reported.
• Use Misleading Graphing Scales – What better way to show a company’s strengths then to present it visually.  Many people are more likely to be attracted to visual evidence than to reading the full report.  But what is often hidden by visuals is the real story.  Presenting information in a graph, such as a line or bar graph, is affected by the scale used to represent the information.  This can be seen frequently on financial television shows or on websites where a quick look at a company’s stock price shows what appears to be a big drop.  Yet the scale used is actually in very small increments and not really representative of the percentage change which may be quite small.  For market research, how scales are used to represent change (e.g., sales, market share, customer service rating, etc.) is very important and should be consistent between different analyses.  For example, the graphing scale used to compare how the company is perceived by their customers on one characteristic should be the same scale used to show other characteristics.  Reports that continually show different increments on graphing scales are probably doing so to make something appear important when in fact it is not or worse, hiding something that will present the industry, company or product in a bad light.

One final consideration on the report presentation.  Some researchers will write their report in a manner that focuses only on the results and does not attempt to offer explanations for the results.  In these cases the reasons for the results are left up to the reader.  However, other researchers will go a step further and offer explanations for the results, often offering their own opinions based on conclusions they have drawn from the research.  Of course, their conclusions may be completely wrong if the research was performed poorly.  In any research where the report includes “implications” of the research the reader should not automatically take these as truth.  Only after careful analysis of the research design can the value of these implications be considered.