Respondent collaboration in web survey design: Developing a unique strategy for pre-testing questions and improving the quality of Internet survey research.

摘要

A new form of survey pre-testing, which capitalizes on the speed, fluidity and accessibility afforded by computer-meditated communication, was investigated through a two-stage experimental procedure. The first stage involved a representative sample (N = 284) from an organization participating in a Web-based exercise that allowed them to select a series of questions which they believed should be used in a future survey of their fellow members. Also during this stage, a panel of three experts in the field of survey methodology similarly chose a series of questions from the same pool of prospective items. Two Web surveys were then constructed based on the results of this first stage. The second stage involved participants from the same organization (N = 1909) being randomly assigned to either the respondent-oriented (i.e., experimental) survey or the survey derived entirely by the expert panel (i.e., the control survey). Results showed that subjects taking the experimental survey were significantly more likely to report that the questions covered topics and issues that were more personally relevant and pertinent to the organization. Completion rates were also significantly greater for the experimental survey, (by approximately 2%). Subjects taking the experimental survey were also more likely to provide answers that could be construed as more "committed" and less equivocal when expressing their opinion compared to control subjects. The experimental survey also demonstrated more predictive validity, although not by margins large enough to satisfy robust confidence intervals. This research reveals how this experimental form of pre-testing represents a new wave of opportunities for harnessing the Internet to improve survey research methods, and the benefits of employing paradigms such as user-centered design, as well as grounded and personal construct theories, to achieve these ends.