基于MapReduce的术语权重计算方法研究

摘要

Term recognition is widely used in the ontology construction, dictionary construction and other fields. And term weighting is a key step in the term recognition. In this paper, several improvements have been made to TF-IDF algorithm, e.g., the length of terms is considered in weighting, also with terms' correlations to documentation set. The candidate term weight is calculated in a distributed manner based on MapReduce on Hadoop. Experimental results show that the method proposed not only simplifies the steps of term weighting, but also improves the efficiency of the algorithm.%术语识别在本体构建、词典构建等领域应用广泛,而术语权重计算是术语识别中的关键步骤.本文通过改进TF-IDF公式,将组成术语词条的长度作为权重因素之一,同时考虑术语在文档集中的领域相关性.整个过程基于MapReduce编程模型实现,在Hadoop云平台中以分布式方式计算候选领域术语的权重.实验结果表明,该方法不仅简化了术语权重计算的实施步骤,也提高了算法执行效率.