How to Evaluate Market Research Reports

3. What was used to measure responses?

Survey research requires the use of a questionnaire or survey “instrument” for collecting information from respondents.  Yet questionnaires can be easily flawed and lead to serious validity problems (i.e., is it measuring what it is supposed to measure).  When examining the results of respondents’ answers to survey questions, it is important to recognize that poorly structured questions can lead to biased results.  Here are a few ways questions can be poorly written:

• Leading Questions – These questions are biasing since they are written in such a way as to suggest an answer to the respondent (e.g., “Wouldn’t you agree that our product has better overall performance than our competitor’s product?”).  In this case the question spins a positive “Wouldn’t you agree” instead of using a more neutral statement (e.g., “How do you rate the overall performance of our product compared to our competitor’s product?”).
• Loaded Questions – These questions provide information in the wording that plays on a respondent’s emotions or are slanted in a particular way that can effect a respondent’s judgment (e.g., “Do you believe that our keen knowledge of what is important to the market allows us to respond better to customers’ needs than our competitor?”).  Clearly the inclusion of “our keen knowledge of what is important to the market” paints a picture that can bias a respondent’s perception of the company.
• Double-Barreled Questions – These questions contain more than one issue within a question thus making it difficult to determine what is being measured (e.g., “How would you rate the speed and durability of our product?”).  Both the “speed” issue and the “durability” issue should be evaluated with two separate questions.
• Ambiguous Questions – These questions contain words that are either too general or that respondents may interpret in different ways (e.g., “Do you regularly use our product?”).  In this example the word “regularly” could be interpreted by one respondent as being everyday while another may say yes even though they only use the product once a month.

Additionally, questions about the validity of the research increases if the report does not include the actual survey questions used to obtain the results but instead states the general nature of a question, such as “When asked about their preference for different brands, customers responded... ” Clearly if the actual question was tainted with any of the four issues discussed above then the results would be suspect.

Professional market researchers are well-schooled in how to prepare good questions and are able to test these prior to making them available to all.  But testing questions is expensive and takes great knowledge of research to do it correctly.  Consequently, most report generators who are not accustomed to doing market research will never go down this road.