基于网络爬虫和改进的LCS算法的网站更新监测

摘要

With the explosive growth of information in Internet era, customers need to get the required information conveniently and timely.Traditional search engine and website subscription represented by RSS couldn't satisfy the users' high equality demand due to their disadvantages.Based on this, a new universal method of website update detection and subscription based on web crawler and text contrast is proposed.After analyzing and processing the successive webpage content, the proposed method would contrast the text, update contents and return the structured results to users.Experiment results show that this method conquers the difficult of RSS subscription's feed limit, and makes it possible to add subscription on one's own.It also got good performance upon university, enterprise, news, video, blog, forum websites etc.%互联网时代,信息爆炸式增长,用户需要方便及时地获取自己所需的信息.传统的搜索引擎和以RSS为代表的订阅具有一些缺陷,难以满足用户高质量需求.在此基础上,利用网络爬虫和文本对比,提出一种新型网站更新监测与订阅的通用方法.该方法将先后抓取的网页内容分析处理后,进行文本对比,检测更新内容,将结果以结构化形式返回给用户查看.实验表明,该方法解决了RSS订阅受订阅源限制的缺点,实现了用户添加任意网站,在高校、企业、新闻、电影、博客、论坛等网站的监测方面具有较好的效果.