Determining Evaluation Criteria for a 'Seal of Approval' Process for Probabilistic Risk Assessment Analysis Tools

摘要

"Bring me a rock." This phrase refers to the practice of making an oftentimes vague request for material or information - in our case, for evidence that a computer-based analytical tool is acceptable for use in a given domain -rejecting what's being proffered because it's just not "quite right," and sending out for replacements in similarly vague terms. It's a situation potential members of a proposed NASA working group would like to avoid. The group, if approved, will serve as a clearinghouse for analytical tools used in probabilistic risk assessment. The potential members want to have a well-ordered, documentable approval process using appropriate evidence. They'd like to be able to specify up-front the evidence they expect to receive from vendors and developers to facilitate the approval process. They also want to ensure that all analytical tools brought to them for approval for use receive appropriate levels of inspection and evaluation. They'd also like the process to be auditable and repeatable. None of these expectations permits the existence of a "bring me a rock" situation. This paper outlines initial work performed in the design of an approval process to meet the above expectations, and initial work performed to define appropriate evidence to use in the process. We first discuss the approval process in general to provide the framework for the discussion of the primary focus of the paper, the presentation of work performed to design and populate a proposed decision support tool, the Fitness for Use Worksheet. The worksheet documents an expert assessment of an analysis tool's fitness for use in a selected domain based on the acceptability and importance of presented evidence. We conducted a survey eliciting expert opinion on evidence typically used in similar approval processes as well as evidence that should be used in an effort to define evidence to be used in the proposed approval process. We also elicited expert opinion as to the relative importance of evidence. The gathered information will be used in the initial population of an "evidence database" that will support the worksheet. The wide variety of survey responses as to the importance and use of different evidence reflects, for example, the heterogeneity of the represented communities. We conclude by discussing a potential development plan for the approval process.