The purpose of this paper is to test the reliability of query intents derived from queries, eitherby the user who entered the query or by another juror. We report the findings of three studies:First, we conducted a large-scale classification study (approximately 50,000 queries) using acrowdsourcing approach. Then, we used click-through data from a search engine log andvalidated the judgments given by the jurors from the crowdsourcing study. Finally, weconducted an online survey on a commercial search engine’s portal. Since we used the samequeries for all three studies, we were able to compare the results and the effectiveness of thedifferent approaches, as well. We found that neither the crowdsourcing approach using jurorswho classified queries originating from other users, nor the questionnaire approach usingsearchers who were asked about their own query that they just entered into a web searchengine, lead to satisfying results. This leads us to conclude that there is little understanding ofthe classification tasks, even though both groups of jurors were given detailed instructions.While we used manual classification, our research has important implications forautomatic classification, as well. We must question the success of approaches usingautomatic classification and comparing its performance to a baseline from human jurors.
展开▼
