An Improved Bloom Filter in Distributed Crawler

摘要

Distributed crawlers have brought great value in both business and scientific research by crawling online data resources, while a large number of duplicate url links seriously affect the efficiency of crawlers. The bloom filter represents the set through an array of bits and uses the hash function to query the elements, which improves the efficiency of query data when space utilization is low. However, generating false positive is an inevitable problem for bloom filter. In this paper, the MD5 algorithm is used to pretreat the URL, and an improved multi-dimensional bloom filter algorithm is proposed, which effectively reduces the rate of false positive and improves the efficiency of distributed crawler.