Effective result inference is an important crowdsourcing topic as workers may return incorrect results. Existing inference methods assign each task to multiple workers and aggregate the results from these workers to infer the final answer. However, these methods are rather ineffective for context-sensitive tasks (CSTs), e.g., handwriting recognition, due to the following reasons. First, each CST is rather hard and workers usually cannot correctly answer a whole CST. Thus a task-level inference strategy cannot achieve high-quality results. Second, a CST should not be divided into multiple subtasks because the subtasks are correlated with each other under certain contexts. So a subtask-level inference strategy cannot achieve high-quality results as it neglects the correlation between subtasks. Thus it calls for an effective result inference method for CSTs. To address this challenge, this paper proposes a smart assembly model (SAM), which can assemble workers' complementary answers in the granularity of subtasks without losing the context information. Furthermore, we devise an iterative decision model based on the partially observable Markov decision process, which can decide whether we need to ask more workers to get better results. Experimental results show that our method outperforms state-of-the-art approaches.
展开▼
