Construct Validity and Generalizability of Simulation-Based Objective Structured Clinical Examination Scenarios

摘要

What was known Objective Structured Clinical Examinations (OSCEs) are increasingly used to assess the competence of residents, including both formative and summative high-stakes assessments.;What is new A set of simulation-based OSCE scenarios has construct-related validity and generalizability in the assessment of residents at different levels of training.;Limitations Single-site study, lacking a formal power calculation; use of advancing clinical year as the sole discriminant variable for competence.;Bottom line Reduction in error rates, and progressive score and performance for residents with advancing training level demonstrate construct-related validity and utility of OSCE scenarios.;Editor's Note: The online version of this article contains 6 appendixes: Israeli examination process development and adaptation; process validation of a simulation-based OSCE used in Israeli Anesthesiology Board Examination; examples of operating room, trauma, and resuscitation scenario checklists; examinee and examiners feedback; and calculated checklist formulas.;Introduction The Israeli Board Examination in Anesthesiology simulation-based objective structured clinical examination (OSCE) is well described and validated1–3 using objective and subjective parameters.2–4 The examination is administered only to graduating residents, and its construct-related validity (progression of OSCE scores across years of training)4 has not been evaluated, as all Israeli participants were at the same level of training. Since 2004, the National Board of Medical Examiners has required a simulation-based clinical skills examination for medical students.5 The Accreditation Council for Graduate Medical Education Outcome and Milestone Projects6–9 led the American Board of Anesthesiology (ABA) to give diplomates enrolled in Maintenance of Certification in Anesthesiology from 2000 through 2007 the option to complete an endorsed simulation course to support their knowledge and skills.10 More recently, the ABA implemented a simulation-based OSCE as part of its certification examinations.11 As development of simulation-based OSCEs carries time and financial costs, sharing of validated scenarios across nations would facilitate their broader dissemination and implementation. We assessed the construct validity of simulation-based OSCE summative assessment tools developed for the Israeli Board Examination, and their potential generalizability to a US training program for formative or summative assessment. We examined anesthesiology residents across all postgraduate years (PGYs) 2 to 4 at 1 institution. This validation could not be performed in the Israeli Board setup, which tested only graduating residents (equivalent to US PGY-4 residents). The other aim was to demonstrate the generalizability, using scenarios developed for an international examination in a US academic environment for formative (teaching) and summative (testing) assessment.;Methods The study was conducted at the University of Florida Anesthesiology Residency Program. In a simulated environment, 2 similar but not identical scenarios (to counter scenario content leakage and enhance content security) were used in each of 3 clinical domains: resuscitation, trauma, and operating room crisis management. The scenarios were originally developed by the Israeli Board of Anesthesiology Examination Committee.2,3,12,13 Faculty members from the Department of Anesthesiology at the University of Florida, assisted by education and simulation experts, translated the scenarios and assessment tools, with maximal adherence to the original script,2,3 scenario protocol, language, and assessment tools. No changes were made in scoring, assessment, pass/fail determinations, orientation of residents, or the examination process itself (see appendixes 1 and 2 for development and validation). Individual performance in each scenario was assessed by using a 12- to 20-item scenario-specific checklist. Checklist items covered simple and