Effectiveness of a Traditional Gradesheet for Measuring Air Combat Team Performance in Simulated Distributed Mission Operations

摘要

The utility and effectiveness of simulation-based training of air combat skill has been supported by a long research history. However, despite this success, there has been much debate on the effectiveness of gradesheets that have been used as the primary measure of air combat skill. This study assessed the effectiveness of a Gradesheet that has been used to measure air combat team performance in Distributed Mission Operations (DMO). It was hypothesized that the DMO Gradesheet is an effective measure of air combat team performance, and is therefore sensitive to differences in team performance over time during training, among performance indicators, and across team experience levels. Between August 2000 and December 2001, 32 teams of F-16 pilots from various U.S. Air Force operational units participated in a week of structured DMO training. Training consisted of teams flying four networked high-fidelity F- 16 simulators in scenarios against multiple constructed threats. On all scenarios of the training syllabus, air combat Subject Matter Experts (SMEs) graded the performance of the teams on 40 indicators that make constitute the DMO Gradesheet. Analyses of the data found that aggregate mean graded team performance increased over a week of DMO training. However, for all indicators, mean graded performance increased linearly and at approximately the same rate, suggesting that the Gradesheet captured only a single component of air combat performance. Furthermore, team experience did not moderate change in graded performance over missions as expected, undermining the claim that the Gradesheet measured even general performance. Results suggest that the DMO Gradesheet as used in the current study lacked the sensitivity, validity, and reliability desired in a measure of air combat performance. Future directions in the development of subjective and objective measures of air combat performance are discussed. (8 tables, 21 figures, 24 refs.).