大数据下基于多CPU的两级指纹流水计算去重方法

摘要

This paper analyzes the importance of data removal. According to the existing data of the removal techniques and algorithms, it improves MD5 code fingerprint algorithm to calculate and optimize it, analyzes and recombines the fingerprint calculation of water level. Using the cache group to replace the single cache, we propose a new method to calculate the two water level fingerprints based on multi CPU to study and analyze the method. At the same time, it supports effectiveness of the method through the relevant tests and test data.%分析数据去重的重要意义, 根据现有的数据去重技术和算法, 改进 MD5 码指纹的计算算法并进行优化,分析并重组指纹计算的流水化方法, 利用缓存组代替单个缓存的方式, 提出一种基于多 CPU 的两级指纹流水计算方法, 对该方法进行分析研究, 并通过相关试验和试验数据来支持该方法的有效性.