An Investigation of Reference Ratings of Applicants for Administrative Law Judge

摘要

The purpose of this study was to analyze the reference ratings of applicants for administrative law judge to determine the reliability and other psychometric properties of the ratings and their relationship to certain rater characteristics. The analysis was conducted on the ratings made by 883 raters of 58 applicants. Each applicant was rated on 27 different abilities and traits. The results indicated that the applicants were rated very favorably, almost all of them were evaluated as better than average or outstanding. Despite the small discriminations of the total ratings among applicants, the average interrater reliability was estimated to be .75. A mathematical averaging of all the ratings obtained by each applicant yielded approximately the same overall evaluation of applicants as a subjective averaging of the ratings by personnel examiners. Overall evaluations of applicants based on certain subgroups of raters did not agree well with overall evaluations made by personnel examiners. The nature of the relationship and the frequency of contact between rater and applicant determined, to some degree, how favorably the applicant was rated.