面向Web论坛的网络信息获取技术及系统实现

摘要

网络爬虫技术是网络信息获取的重要手段,面向Web论坛的信息获取则是网络爬虫技术所面临的新课题.在分析和研究面向Web论坛信息获取技术的基础上,本文设计和实现了一种用于Web论坛信息获取的主题网络爬虫系统,根据Web论坛信息组织结构,提出了基于遍历策略的信息搜索技术;根据正文信息分布及论坛自身特点,提出了基于DOM与分块算法相结合的正文提取技术.实验结果表明,遍历策略比传统的网络爬虫遍历策略具有更高的效率,能够采集到更多主题相关度高的网页;经过噪声清洗处理后,有效提取网页正文,提高了信息采集精度.%The Web spider is very important in gathering information, which also faces new challenges when it's been used in crawling the Web forum.This paper mainly studies the basic technologies of crawling in the Web forum, designs and implements such a system, which is mainly used to gather the information of the Web forum.According to the information structure, a traversal strategy is proposed.Based on the distribution of the context, a DOM and block algorithm is proposed.The experimental result shows that the traversal strategy is more efficient than the traditional traverses to get those highly subject-relevant Web pages, and after using the strategy for the context extracting of Web pages, effectively improves the accuracy of the information collection.