基于改进Kademlia协议的分布式爬虫

摘要

With the explosive growth of Internet information, researches on search engine and big data call for an efficient, stable and scalable crawler architecture to collect and analyze Internet data. Inspired by peer to peer network, we use distributed hash table as a carrier of communication between nodes, while a distributed hash table implementation—Kademlia protocol is modified and improved to meet the needs of the distributed crawler cluster's scalability and fault tolerance. In the experiments, we carried out multi-threaded experiment on single computer and node expansion experiment on distributed cluster. From system performance and system load point of view, the experimental results show the effectiveness of this kind of distributed cluster.%随着互联网信息的爆炸式增长,搜索引擎和大数据等学科迫切需要一种高效、稳定、可扩展性强的爬虫架构来完成数据的采集和分析.本文借助于对等网络的思路,使用分布式哈希表作为节点间的数据交互的载体,同时针对网络爬虫自身的特点,对分布式哈希表的一种实现——Kademlia协议进行改进以满足分布式爬虫的需求.在此基础上设计并完善了具有可扩展性和容错性的分布式爬虫集群.在实际试验中,进行了单机多线程实验和分布式集群的实验,从系统性能角度和系统负载角度进行分析,实验结果表明了这种分布式集群方法的有效性.