Experiments in the Wild: Public Evaluation of Off-Screen Visualizations in the Android Market

摘要

Since the introduction of application stores for mobile devices there has been an increasing interest to use this distribution platform to collect user feedback. Mobile application stores can make research prototypes widely available and enable to conduct user studies "in the wild" with participants from all over the world. Previous work published research prototypes to collect qualitative feedback or to collect quantitative attributes of specific prototypes. In this paper we explore how to conduct a study that focuses on a specific task and tries to isolate cause and effect much like controlled experiments in the lab. We compare three visualization techniques for off-screen objects by publishing a game in the Android Market. E.g. we show that the performance of the visualization techniques depends on the number of objects. Using a more realistic task and feedback from a hundred times more participants than previous studies lead to much higher external validity. We conclude that public experiments are a viable tool to complement or replace lab studies.