A decentralized redundancy generation scheme for codesrnwith locality in distributed storage systems

摘要

The increasing data volume in a large number of applications presents a dire need for supportingrnthe reliable datamanagement in distributed storage systems. Existing classical erasure codes,rnsuch as the Reed-Solomon codes and locally reconstruction codes, are widely adopted by manyrndistributed storage systems. However, existing researches mainly focus on proposing new optimizedrncodes, ignoring the optimization of the encoding process with the classical codes, whererninefficient encoding process greatly degrades the encoding performance of the distributed storagernsystems. Thus, how to complete the encoding process in an efficient way has become thernchallenge for adopting the classical codes. In this paper, we propose a decentralized redundancyrngeneration scheme on the basis of the codes with locality, called D2CP, where a 2-step frameworkrnis proposed to support both the data patterns (replication to encoding and direct encoding) andrncodeswith locality with any parameter set. For improving the insertion throughput,D2CP adoptsrna data placement techniquewith consistent hashing to guide the selection of nodes. For reducingrnthe network traffic cost,D2CP adopts a data sending scheduling technique to schedule the transmissionrnof the source nodes and a cooperative parity generation technique to generate the parityrndata cooperatively. To evaluate the performance of D2CP, we conduct experiments on our RAIDrndistributed storage system under various parameter settings with both 30 physical and 200 virtualrnservers. Extensive experiments confirm that D2CP can improve the encoding throughput byrn20% and 32% and reduce the network traffic cost by 16% and 33% compared with the typicalrnapproaches on average for the 2 data patterns respectively