Training Emphasis Task Factor Data: Methods of Analysis

摘要

REXALL, a program within the Comprehensive Occupational Data Analysis Programs (CODAP) system, is routinely used for assessing the level of interrater agreement obtained when multiple raters evaluate 'training emphasis' at the task level and for extracting a reliable common rating policy (CRP). For some samples, very poor interrater agreement precludes extraction of a reliable CRP and limits use of the data. Since poor interrater agreement may be a function of multiple rating policies, research was initiated to develop a methodology for identifying the multiple rating perceptions that may exist within task factor data. The findings presented include the effect of sample size on interrater agreement and the use of modified REXALL analysis, cluster analysis and factor analysis techniques for identifying multiple rating policies in training emphasis data. Results indicate that REXALL analysis employing new CRP extraction criteria is adequate for samples where the CRP includes all raters and when the CRP has a divergency of less than 25%. It was also found that principal components factor analysis has high utility for identifying the CRP and any other rating policies that might exist in the rating data. Possible causes of poor interrater agreement and several alternative approaches to identifying the causes for interrater disagreement are discussed, Guidelines for occupational analysts to follow when using the REXALL and alternative analysis procedures are provided.