Twenty Years of Experience with Expert Judgments

摘要

Over the last twenty years the Delft University of Technology has developed methods and tools to support the formal application of expert judgement. About 1,000 experts have been elicited, providing data for almost 5,000 query variables (almost 100,000 elicited questions in total). 23 Expert judgement panels were used, consisting of 273 experts. Applications were made in a diversity of sectors. The techniques developed at TUDelft can be applied to give quantitative assessments or qualitative and comparative assessments. The former give rise to assessments of uncertainty in the form of probability distributions, from which nominal values of parameters can be derived for practical applications. The latter lead to rankings of alternatives. The overall goal of these formal methods is to achieve rational consensus in resulting assessments. This requires that diverse stakeholders buy into the process by which the results are reached, and that the process itself optimises performance, as measured by valid performance criteria. Performance criteria are based on control assessments, that is, assessments of uncertain quantities, closely resembling the variables of interest, for which true values (e.g., from experiments) are known post hoc. A Procedure Guide for Structured Expert Judgement is published by the European Commission as EUR 18820. The paper explains the experience gained with the expert judgment techniques applied by the TUDelft team.