How robust are conclusions from a complex calibrated model, really? A project management model benchmark using fit-constrained Monte Carlo analysis

摘要

System dynamics-based simulation models are useful for analyzing complex systems characterized by both large parameter spaces and pervasive nonlinearity. Unfortunately, these characteristics also make confidence intervals for model outcomes difficult to assess. Standard Monte Carlo testing with a priori realistic parameter variations produces simulated behavior that is a posteriori improbable, rendering simple Monte Carlo approaches inappropriate for establishing confidence intervals. This paper gives a case study of a model used to forecast completion of design and construction of a large defense program, and proposes a more correct Monte Carlo process, the fit-constrained Monte Carlo analysis. A confidence interval for outcome is computed, using Monte Carlo trials and discarding combinations that do not achieve an acceptable fit of simulated behavior to historical data. For this case, the experiment confirmed the intuitive view that a well-formulated closed loop model calibrated against sparse but widespread data and an appropriate statistical fit criterion can create tight confidence intervals on some model outcomes. By contrast, conventional (non-fit constrained) Monte Carlo results give substantially misleading implications for a confidence interval. The correlations between model parameters and outcomes are also explored, but they do not reveal significant issues with the method or results.