Looking Back on the Past: Active Learning With Historical Evaluation Results

摘要

Active learning is an effective approach for tasks with limited labeled data. It samples a small set of data to annotate actively and is widely applied in various AI tasks. It uses an iterative process, during which we utilize the current trained model to evaluate all unlabeled samples and annotate the best samples based on a specific query strategy to update the underlying model iteratively. Most existing active learning approaches rely on only the evaluation results generated by the current model and ignore the results from previous iterations. In this paper, we propose using more historical evaluation results which can provide additional information to help better select samples. First, we apply two kinds of heuristic features of the historical evaluation results, the weighted sum of historical results and the fluctuation of the historical evaluation sequence, to improve the effectiveness of active learning sampling. Next, to further and more globally use the information contained in the historical results, we design a novel query strategy that learns how to select samples based on the historical sequences automatically. Our proposed idea is general and can be combined with both basic and state-of-the-art query strategies to achieve improvements. We test our approaches on two common NLP tasks including text classification and named entity recognition. Experimental results show that our methods significantly promote existing methods.