Exploiting Pipelined Encoding Process to Boost Erasure-Coded Data Archival

摘要

This paper addresses an issue of , where () erasure codes are employed to archive rarely accessed replicas. The traditional process neither leverages the existence of replicas, nor handles encoding operations in a decentralized manner. To overcome these drawbacks, we exploit pipelined encoding processes to boost the data archival performance on storage clusters. First, we propose two data layouts called and by applying a chained-declustering mechanism to both and redundancy groups. Second, in light of the and layouts, we design two archiving schemes named and , which exhibit the following three - alient features: (i) exploiting data locality—two or three local blocks are read by each involved node for encoding; (ii) decentralized computation load—encoding operations are distributed among nodes; and (iii) parallel archival processing—two or three encoding pipelines are simultaneously deployed to generate parity blocks. We implement both the and schemes and three existing solutions (i.e., SynE, DE, and RapidRAID) in a real-world storage cluster. Experimental results show that our archival schemes outperform the other three solutions in terms of archiving time by a factor of at least 3.41 in a nine-node storage cluster. The experiments strongly indicate that the performance bottleneck of SynE lies in its block-receiving stage; it is disk I/O rather than network traffic that dominates archiving time for both the DE and RapidRAID schemes.