基于声学指纹的海量MP3文件近似去重方法

摘要

Song re-uploading had been shared wastes network bandwidth and storage,which needs to use data de-duplication technology.However,the current approach to de-duplication based file bit-feature does not recognize the same song after signal processing or compression.Aiming at this problem,this paper proposes a near de-duplication method of massive MP3 files based on acoustic fingerprint.It combines the certainty of message digest with the robustness of the acoustic fingerprint,after Bloom Filter(BF) de-duplicate data based on the message digest,then reduces acoustic fingerprint for the secondary near de-duplication based on the dimensionality.It ensures efficient at the same time,greatly improves the de-duplication ratio.Experimental results show that this method can improve the de-duplication rate by one time than Content-defined Chunking(CDC) method,and has good extensibility.%在互联网中重复上传他人已经分享的歌曲会消耗网络带宽,浪费存储空间,但目前的重复数据删除方法主要基于文件的二进制特征,无法识别经过信号处理或压缩后的歌曲.针对该问题,提出一种基于声学指纹的海量MP3文件近似去重方法.结合文件消息摘要的确定性与声学指纹的鲁棒性,在采用布隆过滤器对文件消息摘要一次去重的基础上,根据降维后的声学指纹值进行二次近似去重,保证高效的同时提高去重率.实验结果表明,与可变分块检测方法相比,该方法的去重率可提高1倍以上,扩展性较好.