Looking for Answers: Identifying Search Behavior and Improving Knowledge-Based Data Quality in Online Surveys

摘要

Internet surveys are cost-effective and convenient, but give respondents the opportunity to search the Web for answers to factual questions. Surprisingly, few have investigated the quality of political knowledge data collected in crowd-sourced Internet samples. Using a novel technique to identify individual "cheaters'' across six data sets-collected on Amazon's Mechanical Turk (professionalized paid "workers'') and Survey-Monkey (less-professionalized unpaid "audiences'')-we find that cheating is pervasive. In addition to detecting "cheaters,'' we also show that their political knowledge is significantly inflated compared with non-cheaters, posing problems for the accurate measurement of knowledge. Fortunately, we also demonstrate that question-wording alterations can significantly improve data quality and show that cheating is less-common on mobile survey-taking platforms (compared with desktop/laptop computers).