Advanced Topics in Calculating and Using Confidence Intervals for Model Validation

摘要

A confidence interval is an interval estimate of a parameter of a population, such as a mean, calculated from a sample drawn from the population. In addition to its endpoints, a confidence interval has an associated confidence level, which is a statistically justified degree of confidence that the interval actually contains the population parameter. Confidence intervals are often used in model validation. The model to be validated is executed multiple times; those executions compose a sample from the population of all possible executions of the model. A confidence interval is calculated from the results of the model executions as an estimate of the model's response variable that would be found if all possible model executions had been run. If the known or observed value for the simuland corresponding to the response variable is within the confidence interval, or within some acceptable tolerance of its endpoints, the model is considered to be valid for the variable in question. This paper is a continuation of a Fall 2012 Simulation Interoperability Workshop paper; that earlier paper was an introductory tutorial and survey on the calculation and use of confidence intervals for model validation. This paper covers three advanced topics in the same area. The first is a useful quantification of the notion of "close enough" with respect to confidence interval inclusion. The second is a confidence interval adjustment applicable when multiple potentially non-independent model response variables are being validated. The third is the calculation of confidence intervals for the difference of two means. For all three of these topics, the explanations are motivated and illustrated with examples from the literature of their practical application in model validation.