Option weights should be determined empirically and not by experts when assessing knowledge with multiple-choice items

摘要

Multiple-choice tests are frequently used in personnel selection contexts to measure knowledge and abilities. Option weighting is an alternative multiple-choice scoring procedure that awards partial credit for incomplete knowledge reflected in applicants' distractor choices. We investigated whether option weights should be based on expert judgment or on empirical data when trying to outperform conventional number-right scoring in terms of reliability and validity. To obtain generalizable results, we used repeated random sub-sampling validation and found that empirical option weighting, but not expert option weighting, increased the reliability of a knowledge test. Neither option weighting procedure improved test validity. We recommend to improve the reliability of existing ability and knowledge tests used for personnel selection by computing and publishing empirical option weights.