What Is the Best Size for Matrix-Style Questions in Online Surveys?

摘要

Across two studies, we aimed to determine the row and column size in matrix-style questions that best optimizes participant experience and data quality for computer and mobile users. In Study 1 (N = 2,492), respondents completed 20 questions (comprising four short scales) presented in a matrix grid (converted to item-by-item format on mobile phones). We varied the number of rows (5, 10, or 20) and columns (3, 5, or 7) of the matrix on each page. Outcomes included both data quality (straightlining, item skip rate, and internal reliability of scales) and survey experience measures (dropout rate, rating of survey experience, and completion time). Results for row size revealed dropout rate and reported survey difficulty increased as row size increased. For column size, seven columns increased the completion time of the survey, while three columns produced lower scale reliability. There was no interaction between row and column size. The best overall size tested was a 5 x 5 matrix. In Study 2 (N = 2,570), we tested whether the effects of row size replicated when using a single 20-item scale that crossed page breaks and found that participant survey ratings were still best in the five-row condition. These results suggest that having around five rows or potentially fewer per page, and around five columns for answer options, gives the optimal survey experience, with equal or better data quality, when using matrix-style questions in an online survey. These recommendations will help researchers gain the benefits of using matrices in their surveys with the least downsides of the format.