Correct Use of Subject Matter Experts in Cost Risk Analysis

摘要

Subject Matter Experts (SMEs) are commonly used in cost risk analysis for values that either are not available in historical data or for which no appropriate analogy can be found. Problems commonly arise in two areas in particular: (1) when multiple experts give opinions on a single effect or entity and the inputs are not identical in distribution (which is almost inevitable), and (2) when a single expert provides distributional information that is intractable or suspiciously unlikely in its form (which is common). This paper will put forward correct solutions in case (1), in which the authors' experience shows that practitioners (and even experts) use incorrect solutions. It is important to note that the commonly exercised incorrect solution underestimates the dispersion, and thus the 80th percentile, in some cases by a large margin. The authors believe that their solution is rare and, further, are unaware of any use of the solution, and will recommend tenets to guide the practitioner. In preparation for the solutions laid out above, the authors will first describe the method of expert-based risk analysis, with the erroneous method for combining SME testimony, and then show the correction. An analytical treatment will quantify the impacts of the erroneous approach. The paper also will explain why the new method of conflating expert assessments is to be preferred to the common Delphi technique, which may fall prey to both anchoring and domination by a vocal minority. The paper also will briefly address case (2) by presenting common examples of problematic formulations and proposed resolutions. These include intractable specification of a triangular distribution, specification of a discrete categorical distribution when triangular was intended, and specification of a triangular with low and high values that are not the true extremes as well as errors committed by the risk analyst. The presentation includes 51 briefing charts.