The Research and Implementation of a Distributed Crawler System Based on Apache Flink

摘要

Web information is growing at an explosive rate. The crawling ability of the single-machine crawler becomes the bottleneck, so distributed web crawling techniques become the focus of research. However, the existing distributed web crawler systems have some shortcomings. Thread management for solving thread synchronization and resource competition is usually designed by using pure multi-thread asynchronous methods. But the execution of this mechanism observably reduces the performance. Moreover, the deduplication algorithms lead to low efficiency in dealing with large data sets or the problem of occupying large storage space. Therefore, we propose and implement a distributed web crawler system based on Apache Flink, which combines and integrates the Mesos/Marathon framework. It can make full use of the computing resources of the cluster and significantly improve the efficiency of the web crawler system. Taking the data of Netease news pages as an example, the experimental results show that the distributed crawler proposed has higher execution efficiency and reliability.