The traditional high-performance deduplication systems are focused on improving the data compression ratio and throughput of a single node, but most of them have ignored the scalability which makes it difficult to be applied to large-scale distributed environments. This paper presents LevelStore, a large scale key-value store for deduplication storage system which optimized three indicators including compression ratio, throughput and scalability. LevelStore introduced a hierarchical feature which improved the disk utilization by introducing container compaction and exploited a sorted in-memory buffer to favor large sequential writes. More importantly, LevelStore can be dynamically and cost-effectively scaled up on demand. Experiments for different datasets are presented and the results show LevelStore can not only greatly improved the performance of a single node but also provided a lot of efficient building blocks for distributed implementation and thus applicable to large scale and distributed systems.
展开▼
