Research on Optimization of Big data Storage Structure in Distributed System

摘要

In a distributed system, the storage structure of the data directly affects the storage efficiency and processing performance of big data. In the row storage structure, the data from the local read, loading speed, but the compression efficiency is low and there is data redundancy; in the column storage structure, the data compression efficiency is high, but the data cross-node access increased network transmission consumption The Aiming at the shortcomings of the row storage structure and the column storage structure, a kind of storage method combined with rows and columns is proposed to improve the data storage structure. The experimental results show that the improved data storage structure is slightly lower than the row storage in the loading speed. In the data compression, the efficiency of the parallel storage and the column storage is high. The combined storage structure not only avoids the extra disk I/O overhead, but also reduces the unnecessary storage of the network, which greatly improves the storage efficiency and processing performance of the distributed system for big data.