Distributed Mission Operations Within-Simulator Training Effectiveness Baseline Study. Volume 3. Real-Time and Blind Expert Subjective Assessments of Learning

摘要

The current work documented in this report is the subject matter expert (SME) rating data from a large study examining the within-simulator learning benefits of Distributed Mission Operations (DMO) training as described in AFRL-HE-AZ-TR-2006-0015 Vol I, Distributed Mission Operations Within- Simulator Training Effectiveness Baseline Study: Summary Report. That study examined 76 participating F-16 four-ship teams in week-long DMO training exercises and compared beginning-of-week to end-of-week performance on mirror- image air combat scenarios. As a major assessment component to the overall study, the current work reports data we collected from SME ratings of pilot performance. Two SME rating methods were employed -- real-time ratings (i.e., done during mission execution) and SME ratings captured later using a scientific 'blind' protocol. Comparing SME ratings of performance on the mirror- image scenarios revealed highly significant performance increases as a function of DMO training. SME real-time and blind ratings were both found to be significantly higher for end-of-week scenarios. The fact that the real-time and blind rater results corroborated one another provides strong support that performance greatly improved as a function of DMO training. Lastly, though the real-time ratings are not as scientific as the blind ratings, the fact that the blind ratings revealed very similar results to the real-time ratings greatly increases our confidence for continuing to use real-time ratings to assess performance. Compared to implementing an ongoing blind rating system, this justification to use real-time ratings creates a significant logistical, time, and financial savings for future research.