基于WebDriver的定向网络爬虫设计与实现

摘要

With the development of Web 2.0, the scale of information on the network has been growing explosively. Extracting useful information from the public data on the network has become a new hotspot in the field of data mining. Data acquisition is the first step in data analysis. Web sites in the Web 2.0 era change profoundly both in the content and form, the traditional Web Crawler based on static page analysis faces new challenges. In this paper, we use WebDriver to implement a directed web crawler, which can automatically collect specified web page data, support dynamic tech-nologies such as Ajax, and can identify simple code and bypass crawler detection. Application of the crawler on the national enterprise credit information system achieved good results.%随着Web 2.0的兴起，网络上的海量信息迎来了爆发性地增长。从网络上的公开数据中提取有效信息成为了数据挖掘领域的新热点。数据获取是数据分析的第一步。Web 2.0时代的网站在内容和形态方面发生了深刻的变化，对传统的以静态页面分析为基础的网络爬虫提出了新的挑战。本文利用 WebDriver 实现了一个定向网络爬虫，该爬虫能够自动采集指定网页数据，支持Ajax等动态技术，能够对简单验证码进行识别，并绕过爬虫检测。利用该爬虫对全国企业信用信息系统进行爬取，取得了良好效果。