A decentralized redundancy generation scheme for codeswith locality in distributed storage systems

摘要

The increasing data volume in a large number of applications presents a dire need for supportingthe reliable datamanagement in distributed storage systems. Existing classical erasure codes,such as the Reed-Solomon codes and locally reconstruction codes, are widely adopted by manydistributed storage systems. However, existing researches mainly focus on proposing new optimizedcodes, ignoring the optimization of the encoding process with the classical codes, whereinefficient encoding process greatly degrades the encoding performance of the distributed storagesystems. Thus, how to complete the encoding process in an efficient way has become thechallenge for adopting the classical codes. In this paper, we propose a decentralized redundancygeneration scheme on the basis of the codes with locality, called D2CP, where a 2-step frameworkis proposed to support both the data patterns (replication to encoding and direct encoding) andcodeswith locality with any parameter set. For improving the insertion throughput,D2CP adoptsa data placement techniquewith consistent hashing to guide the selection of nodes. For reducingthe network traffic cost,D2CP adopts a data sending scheduling technique to schedule the transmissionof the source nodes and a cooperative parity generation technique to generate the paritydata cooperatively. To evaluate the performance of D2CP, we conduct experiments on our RAIDdistributed storage system under various parameter settings with both 30 physical and 200 virtualservers. Extensive experiments confirm that D2CP can improve the encoding throughput by20% and 32% and reduce the network traffic cost by 16% and 33% compared with the typicalapproaches on average for the 2 data patterns respectively