The use of online research panels has for some time received bad reviews from serious marketing researchers who have questioned the validity and reliability of this form of research. Their key complaint has to do with who participates in these panels and what their real motivation is for being part of a panel. This is particularly a problem when online participants are rewarded for their efforts such as receiving cash or other incentives. Now several big marketing companies are pushing to find out how good the information really is that comes from online panels.
While most fill out surveys to be heard, those who take surveys for the money turn out to be less diligent about filling out the survey. Certain research panels tend to give outright cash gifts which attract more of those people.
How can the results of panels cited in the story be consistent when replicating their own research but show inconsistency across panels (i.e., when the results of one panel are compared to another)?