Research of Distributed Data Optimization Storage and Statistical Method in the Environment of Big Data

摘要

Through analyzing the characteristics of the data processing in the environment of big data, the author puts forward the distributed optimization storage and statistical system model based on hash distribution. Based on massive amounts of data distribution, this model completed the data backup protocol based on master copy of distributed design and the Paxos distributed system protocol design on the basis of the principle of consistency distribution of hash algorithm and fully considering load balancing. The author makes simulation of the process of data storage through OPNET Modeler simulation software, and then complete performance evaluation of the designed storage model. The simulation results show that the big data distributed storage system model design based on hash distribution this paper proposed compared with sequential storage distribution policy increases by 12.2% in terms of throughput, and the response latency decreases by 9.8%. In addition, the random writing speed is about 8Mb/s. In general, the model of distributed storage this paper designed based on hash distribution has feasibility.