In God We Trust, All Others Bring Data: A Bayesian Approach to Standard Setting

摘要

Standard setting is an inherent part of pass/fail decisions in assessment. Although various standard setting Method s are available, they all have their limitations and no Method provides a golden solution to all our standard setting headaches. Some Method s require potentially labor-intensive standard setting panels of judges who have specific knowledge. Other Method s require student cohorts of ‘sufficient’ size. However, small cohorts are quite prevalent in medical programs across the globe, and standard setting panels are not always feasible due to logistic or financial constraints or may result in inadequate judgments due to bias or a lack of specific knowledge. This manuscript presents a new standard setting Method , which is based on the Bayesian principle of updating our knowledge or beliefs about a phenomenon of interest with incoming data, uses information that is not considered in Method s already available and can be applied to both small and larger cohorts regardless of whether standard setting panels are available. As demonstrated in this manuscript through a worked example, the new Method is easy to implement and requires only a minimum of calculations which can be done in zero-cost, user-friendly Open Source software. Options for future research comparing different standard setting Method s are discussed.