Comparing the accuracy of experimental estimates to guessing: a new perspective on replication and the “Crisis of Confidence” in psychology

摘要

We develop a general measure of estimation accuracy for fundamental research designs, called v. The v measure compares the estimation accuracy of the ubiquitous ordinary least squares (OLS) estimator, which includes sample means as a special case, with a benchmark estimator that randomizes the direction of treatment effects. For sample and effect sizes common to experimental psychology, v suggests that OLS produces estimates that are insufficiently accurate for the type of hypotheses being tested. We demonstrate how v can be used to determine sample sizes to obtain minimum acceptable estimation accuracy. Software for calculating v is included as online supplemental material (R Core Team, 2012).