有效的爬行Ajax页面的网络爬行算法

摘要

The generation of Ajax web pages and the Ajax page navigation must execute the client JavaScript, thus it is impossible to extract the complete content of an Ajax page through the traditional crawling algorithms. In this paper, the working mode of Ajax is analyzed, the problem of crawling Ajax web pages is elaborated, and an effective algorithm for crawling Ajax pages is proposed. The algorithm can realize the dynamic generation of Ajax web contents in client browser and the navigation of Ajax web pages, and also it can assign identification number for the crawled pages whose static pages can be generated. Experimental result shows that the number of Ajax pages crawled by the proposed algorithm is obvious bigger than the traditional ones’, and the presented replicas-detecting policies can effectively reduce the time consumption of the algorithm.%　　Ajax页面的生成和页面导航需要执行客户端的JavaScript代码，传统网络爬行算法无法获取Ajax页面全部内容。分析了Ajax的工作方式，阐述了爬行Ajax网页所面临的主要问题，提出并实现了一种有效爬行Ajax页面的网络爬行算法。该算法可控制客户端浏览器动态生成页面内容和完成页面导航，为爬行过的页面分配标识编号并生成相应静态页面。实验结果表明，提出的算法所爬行的Ajax页面数量明显多于传统方法，同时，采用的双重消重策略可有效减少算法的时间耗费。