Hadoop下改进布隆过滤器算法的网页去重

摘要

To solve the space waste problem existing in the server space where a lot of duplicated and similar data are stored,we propose an improved Bloom Filter algorithm,which adds an array of bit and dynamically optimizes the number of copies of duplicated data according to the weight calculated by the repeated hits of the bit array.Then,the improved algorithm is parallelized in the Hadoop distributed cluster to further improve the processing efficiency.Experimental results show that compared with traditional web duplicate removal algorithms,the improved Bloom filter algorithm can not only improve the processing efficiency of jobs,but also save the server storage space to a certain extent by dynamically optimizing the number of copies of duplicated data according to the repeated hits of the bit array.%针对服务器中存储的大量重复和相似数据造成的空间浪费问题,改进的布隆过滤器(Bloom Filter)算法通过增加位数组并根据位数组的重复命中次数所计算的权重来动态优化重复数据的副本数,然后在Hadoop分布式集群下对改进的算法进行并行实现,以进一步提高作业处理效率.实验结果表明,与传统网页去重算法相比,改进的Bloom Filter算法的并行实现不仅提高了作业的处理效率,而且通过基于位数组下动态重复次数对副本数的优化,在一定程度上节省了服务器的存储空间.