Optimization Scheme of Massive Meteorological Data Storage Based on OpenStack Swift

摘要

In order to solve the problem that cloud storage system will produce lots of metadata information and reduce the storage efficiency of the system when processing massive meteorological small files. In this paper, we propose an optimized scheme for storing massive meteorological data, which is implemented in OpenStack Swift. The scheme consists of load balancing module and merge storage module. On the one hand, load balancing module combines HAproxy and UCARP to alleviate the uneven load distribution when Swift stores a huge number of meteorological files. On the other hand, the scheme adopts the merge storage strategy to improve the reading and writing performance of meteorological small files in Swift. Experiments show that this optimization method can not only reduce the reading delay of massive meteorological files in Swift system, but also improve the throughput of Swift.