How robust are conclusions from a complex calibrated model, really? A project management model benchmark using fit-constrained Monte Carlo analysis

摘要

System dynamics-based simulation models are useful for analyzing complex systems characterizedby both large parameter spaces and pervasive nonlinearity. Unfortunately, these characteristicsalso make confidence intervals for model outcomes difficult to assess. Standard MonteCarlo testing with a priori realistic parameter variations produces simulated behavior that is aposteriori improbable, rendering simple Monte Carlo approaches inappropriate for establishingconfidence intervals.This paper gives a case study of a model used to forecast completion of design and constructionof a large defense program, and proposes a more correct Monte Carlo process, the fitconstrainedMonte Carlo analysis. A confidence interval for outcome is computed, using MonteCarlo trials and discarding combinations that do not achieve an acceptable fit of simulated behaviorto historical data. For this case, the experiment confirmed the intuitive view that a wellformulatedclosed loop model calibrated against sparse but widespread data and an appropriatestatistical fit criterion can create tight confidence intervals on some model outcomes. By contrast,conventional (non-fit constrained) Monte Carlo results give substantially misleading implicationsfor a confidence interval. The correlations between model parameters and outcomesare also explored, but they do not reveal significant issues with the method or results.