Practical statistical analyses of simulation output data

摘要

It has been our observation that in many simulation studies a large amount of time and money is spent on model development and programming, but little effort is made to analyze the simulation output data in an appropriate manner. As a matter of fact, a common mode of operation is to make a single simulation run of somewhat arbitrary length and then treat the resulting simulation estimates as being the "true" answers for the model. Since these estimates are random variables which may have large variances, these estimates may, in a particular simulation run, differ greatly from the corresponding true answers. The net effect is, of course, that there may be a significant probability of making erroneous inferences about the system under study.

One reason for the historical lack of definitive output data analyses is that simulation output data are rarely, if ever, independent. Thus, classical statistical analyses based on independent identically distributed observations are not directly applicable. At the present time, there are still several output analysis problems for which there is no completely accepted solution, and the solutions that do exist are often complicated to apply. Another impediment to getting accurate estimates of a model's true parameters or characteristics is the computer cost associated with collecting the necessary amount of simulation output data. Indeed, there are situations where an appropriate statistical procedure is available, but the cost of collecting the amount of data dictated by the procedure is prohibitive. We expect this latter problem to become less important as the cost of computer time continues to drop.

Our goal in this talk is to give a state-of-the-art treatment of statistical analyses for simulation output data, and to present the material with a practical focus which should be accessible by a reader having a basic understanding of statistics. The emphasis will be on statistical procedures which are relatively easy to understand and apply, have been shown to perform well in practice, and have applicability to real-world problems.

Most of the material presented in this talk may be found in Law and Kelton (1981).