Generalizability Theory: An Assessment of Its Relevance to the Air Force Job Performance Measurement Project

摘要

Generalizability theory is a method for estimating the dependability of scores over various conditions of measurement. In contrast to classical test theory, which permits the investigation of only one error source at a time, generalizability theory is multifaceted and allows the researcher to propose and simultaneously investigate multiple sources of error. The applicability of generalizability theory to the Air Force's ongoing Job Performance Measurement (JPM) Project is reviewed in this paper. It was concluded that generalizability theory would be relevant to the JPM Project because it is a sophisticated and efficient method of investigating reliability, a fundamental property of any measurement instrument. Next, generalizability theory is illustrated by applying it to data collected from Air Force jet engine mechanics. The question of interest was whether performance scores were generalizable (consistent) over different rating sources (incumbents, supervisors, or peers), rating forms (one of four forms used in the project), or specific items included on any one form. The results indicated that scores were generalizable over both forms and items within forms. However, scores were not generalizable over rating sources. Sources tended to differentially rank ratees, depending on the specific form used. When scores were averaged over rating sources, the resulting mean scores were generalized if at least two rating forms were used. Implications of the results and the applicability of generalizability theory for the Job Performance Measurement Project are discussed. (KR)