What we know about context effects in self-report surveys of well-being: comment on Deaton and Stone

摘要

Concerns about context effects are not just academic. Debates about the strength and robustness of these effects have important implications for the use of well-being measures in applied settings, including policy decisions. For example, Deaton (2012) concluded from evidence like this that there are serious problems in using well-being measures to track the economy over time. Indeed, context effects are often cited as one of the strongest reasons for caution. Schwarz and Strack (1999), after reviewing the evidence for context effects, suggested that their influence may lead one to conclude that reports of SWB seem 'too context-dependent to provide reliable information about a population's well-being' (p. 80). However, much of the body of evidence on which such conclusions are based comes from studies with extremely small samples, with results that often do not replicate when replic-ability is tested directly. Deaton and Stone (2016, this issue) describe one of the first studies to find a moderately large context effect with SWB measures in a large, representative sample. However, just as it is important to know whether effects from previous studies replicate (and why), it is critical to understand why this context effect emerges in these data. This explanation is necessary to clarify how likely this effect is to emerge in other studies, how strongly the effect impacts validity, and whether it can be reduced or eliminated in future designs. All measures-whether they are in the 'soft' scientific disciplines of economics and psychology or in the 'harder' fields such as physics and chemistry-have error. If we were to ignore concepts that were difficult to measure, then we would severely restrict what we know about people and their behaviour, as well as about the physical world. The context effects reported in the article by Deaton and Stone are important both because they help clarify some of the limitations of existing measures and because they help researchers understand the processes by which people make well-being judgements. However, we caution against drawing too strong conclusions about the validity and utility of the measures from this study. Context effects like that reported in the Gallup data are rare, they are typically very small in size, and the field continues to make progress in understanding the processes that underlie them, which can lead to further improvements in measurement in the future. Earlier reports on the data set analysed by Deaton and Stone suggest that context effects can be overcome, and we need to know more precisely how and when this is possible.