基于主题描述模型的相关性判断在网页信息抽取中的应用

摘要

Information extraction from the massive web source is an important way to obtain valuable information and the topic relevant judgment of target web page contents is one of the important steps. At present, manual screening and document training that is the main method for relevance judgment is low efficiency and duplication. In this paper, we attempt to introduce topic description model for measuring relevant analysis. Topic description model measures the topic relevance from the object of task. After the page content analysis, we will weight the document by analyzing the document frequency of the keywords and the change trends of the frequency from the task topic description model for correlation judgment. The experiment verified that the method can effectively improve the efficiency of web information extraction and accuracy, and we get the principle for setting the parameters.%信息抽取是从海量网页获取有价值信息的重要方式,对目标网页内容进行主题相关性判断是提高信息抽取效率和准确性的关键环节.目前的相关性判断主要采用人工筛选和文档训练的方法,这其中存在效率低、重复训练等问题,而本文尝试针对抽取任务引入主题描述模型用于网页内容的主题相关性判断.从任务的主题描述模型的角度出发,计算模型中的关键词基于标记信息的加权频率,将网页内容进行量化表示,然后分析关键词加权频率关于任务主题描述模型的变化来判断网页内容的主题相关性.最后通过对比该方法在国防产品信息抽取中结果,实验证明该方法大大提高了网页信息抽取的效率和准确性.