Distributed deduplication using locality sensitive hashing

摘要

Deduplication in a distributed storage system is described. A deduplication manager defines a first extent on a first node in a distributed storage system. The deduplication manager selects a second extent from the one or more groups of extents. The second extent matches the first extent. The deduplication manager removes, from the first extent, one or more data chunks that are included in the second extent. The deduplication manager associates, with the first extent, a pointer to the second extent for the removed one or more data chunks.