A Self-Matching Sliding Block Algorithm Applied to Deduplication in Distributed Storage System

摘要

The deduplication technology can significantly reduce the amount of storage in data centers, thus to save network bandwidth and decrease the cost of construction and maintenance. Having inspired by the sliding block method of the Sliding Block (SB) algorithm and independent block-dividing thought of the Content Defined Chunking (CDC) algorithm, a Self-Matching Sliding Block (SMSB) algorithm for deduplication is proposed. Via communication with metadata node, the storage system client builds a matching table in local memory that contains fingerprint and checksum, based on the matching table to realize sliding block self-matching so as to detect the duplicate blocks. The experimental results show that the deduplication rate and the disk space utilization rate of SMSB algorithm is respectively 2.03 times and 1.28 times of the CDC algorithm and that the data processing speed is 0.83 times of the CDC algorithm. The SMSB algorithm is suitable for distributed storage system.