An analysis of human factors and label accuracy in crowdsourcing relevance judgments

摘要

Labeling is a complex task. This interesting paper investigates the use of crowdsourcing to create labeled data, which involves relevance judgments that are known to be subjective. The authors use a series of experiments using Amazon's Mechanical Turk (AMT) to explore the human characteristics of the crowd involved in a relevance assessment task. The study considers three factors or variables-the payment offered to the assessor, the effort expended by the assessor, and the assessor's qualifications-and examines the effect of these variables on the resulting relevance labels.