Are all assessments equal? The comparability of demands of college-based assessments in a vocationally related qualification

摘要

The consistency of assessment demands is important to validity. This research investigated the comparability of the demands of college-assessed units within a vocationally related qualification, drawing on methodological approaches that have previously been used to compare assessments. Assessment materials from five colleges were obtained. After some initial familiarisation and the revision of a framework of demands, 15 expert judges compared the assessment materials from pairs of colleges in turn. For each pair, each judge evaluated which college's assessment tasks they considered more demanding on a number of dimensions and overall. Analyses suggested that assessments were more demanding at some colleges than others but that differences were not large. An interview study was also conducted in four colleges. The interview data suggested substantial similarities between colleges in the assessment tasks conducted. However, some differences that could affect demands were identified (e.g., degree of authenticity of tasks, availability of exemplars when writing reports).