Comparing the validity of numerical judgements elicited by direct rating and point allocation: Insights from objectively verifiable perceptual tasks

摘要

Two popular methods for assigning numerical values to a set of to-be-judged objects in order to capture their relative standing are Direct Rating (DR) and Point Allocation (PA). People using PA distribute a fixed sum of 100 points among the objects, while people using DR rate each object on a fixed scale, typically 0-10, later rescaled to sum to 100. Prior research shows that these methods exhibit distinct profiles when values are ranked from largest to smallest, with DR being more test-retest reliable. But which method best translates people's inner judgments into outer numerical values (is more valid)? Instead of examining subjective or abstract stimuli, we use objectively verifiable perceptual tasks, namely judgments of line length presented using bar charts. We show that (i) DR is more inter-rater reliable than PA; (ii) DR is more accurate than PA at the individual level; (iii) but there is no difference in accuracy when individual judgments are combined to form group-level estimates; and (iv) DR judgments were improved by using prior knowledge of method bias, whereas PA judgments were not.