实时Web内容重复识别及排序系统的设计与实现

摘要

To solve the problem that too much similar contents even the same web pages exist in the retrieved web information results,a method to mine the content and structure information of real-time pages as well as the way to recognize its repeats is given. The comparison of document similarity is accomplished based on the PageRank values obtained from the document content and structure information provided by Google. Thc test implemented thc repetition identification and sorting of the real-time webs, and achieved a higher matching degree.%为了解决现有Web信息检索结果中存在较多内容相似甚至相同页面的问题,给出了实时Web内容和结构信息提取的算法及内容重复识别的方法.利用Google提供的PageRank查询接口取得各个页面的PageRank值,结合特定用户的特征信息、查询请求及提取的各Web页内容及结构信息,完成了文档相似度比较,实现了实时页面的重复识别及二次排序,实验结果表明该方法达到了较好的效果.