An exploration of the robustness of some common statistical measures for evaluating air quality models

摘要

The USEPA recommends that specific types and ranges of statistics be used for evaluating regional and urban air quality models. The implementation of these statistical standards implicitly requires a large database composed of observation and model prediction pairs. While the use of such a database may educe the effect of uncertainties in the model predictions and the observations on the computed statistics, it also may mask the true performance skill of a model. In addition, these standards are based on historical model performances and so may not reflect the current capability of air quality models. In this study, the utility and meaning of such standards are explored by comparing ozone predictions from several forecast procedures to observations from the ozone episode of August 3-6, 1990, in the San Joaquin Valley, California, USA. The foecast procedures considered are a SAQM (SARMAP Air Quality Model) simulation, persistence forecasting, random selection forcasting, and constant prediction forecasting. These forcasting procedures range from "acceptable" to "unacceptable" in terms of physical insights. Peak ozone and hourly ozone predictions are evaluated by using 13 statistical quantities which include measures of bias, error, correlation, and scatter and for various subsets of data. We find that these statistics can distinguish among the skills of the persistence, random, and constant forecast procedures. However, when the statistics are applied to the model predictions over all observation/prediction pairs, the resulting values can be misleading as large "nonessential" errors can mask the model skill. The usefulenss of the statistical evaluations suggested by USEPA is brought into question as well by the fact that random and constant forecasts produce "acceptable" normalized biases and normalized gross errors. Suggestions are made for more appropriate applications of statistics for urban and regional air quality model evaluations.