How Close is Close Enough? Statistical Equivalence of Onboard versus Online Surveys of Transit Customers

摘要

Stagnant budgets and growing rates of internet access have increased the appeal of substitutingtraditional survey methods for electronic ones. Online surveys are particularly appealing in the publictransit industry, due to the expense and logistical difficulty of surveying customers onboard buses andtrains. It is therefore critical to understand, quantify, and test the differences between onboard versusonline transit survey data.Traditional hypothesis tests are designed to show that two sample statistics likely come from differentpopulations. However, failing to find a difference cannot be interpreted as evidence that there is nodifference. Furthermore, a difference may be statistically significant, but so small as to provide nopractical insight (which often happens when working with large sample sizes). Statistical EquivalenceTesting (SET) provides an analytical framework with which to evaluate whether two datasets are similarenough to be interchangeable (i.e., statistically equivalent).The paper describes statistical equivalence tests conducted on customer satisfaction data collectedonboard transit systems and data collected electronically via email lists from users of the same systems.We compare proportions of satisfied customers across various economic and travel behaviorcharacteristics between these datasets. Within our chosen threshold of .05 (statistics within fivepercentage points of one another), one of the two datasets evaluated shows strong evidence ofequivalence between onboard and online survey methods, while the other dataset shows strongevidence of nonequivalence. Findings support the idea that, at least in some cases, online surveys cansubstitute for onboard ones.