Workshop session: Maintaining quality

摘要

Compared with independent benchmarks, all four non-probability web surveys gave less accurate results than the probability sample Natsal-3. Comparing results for 35 questions asking about sexual behaviours and attitudes, we found between 60% and 75% of the questions in each of the four web surveys were significantly different from the Natsal-3 benchmarks. Some of the differences were very large - for example, same-sex experience was two to three times higher in the web surveys. Given the similarities in visual presentation between CASI and a web survey, we expected to find fewer differences between the web surveys and the results for questions asked in CASI (in comparison with those asked in CAPI) in Natsal-3. This was borne out by our results. However, there were still significant differences in results between the web surveys and the majority of Natsal-3 CASI questions. This suggests that differences in mode cannot explain all the differences between the web surveys and Natsal-3, and that selection biases must also be present (which is consistent with the results for the demographic questions). Consistent with evidence from other studies, some of our results showed that web surveys obtained higher reports of sensitive behaviours than those in Natsal-3. However, looking at the full range of sensitive behaviours, there was no consistent pattern in whether reports of these were higher in Natsal-3 or in the web surveys. Modifying the quota controls used on two of the web surveys did not lead to consistent improvement in results, and both companies had difficulties meeting the more complex quotas. Of the four web surveys, it was not possible to identify one that performed consistently better than the others. While the demand for web surveys is going to continue to increase, and while the use of volunteer web panels may be suitable for certain types of study, our results support the conclusion that non-probability web surveys are not appropriate if the aim is to provide robust population estimates. Full details and results from our study can be found in Erens, Burkill et al. (2014).