A multi-channel architecture for metadata management in cloud storage systems by binding CPU-cores to disks

摘要

Metadata operations have become dominant file operations in the storage systems. In the scenarios of read-more and write-less of massive small files, the current distributed file systems suffer from the unsatisfying performance and scalability of metadata service because of random disk I/O during metadata operations. In this paper, a highly efficient metadata management architecture for cloud storage systems is proposed. The cluster design significantly improves the scalability of the system. A concept of the disk I/O channel is introduced, which is an independent data storage pipe by binding an independent CPU-core to each physical disk. In addition, a multi-channel fast key-value storage engine is proposed to provide the extremely efficient performance for the underlying storage service, which takes full advantages of multi-core processors and parallel disks I/O. Besides, a new dynamic load-balancing strategy is proposed to reduce load thrashing and improve the precision of rebalancing among the clusters. Performance measurements under a variety of benchmarks show that the metadata management is capable of handling the massive small files storage and the performance is improved significantly compared to the existing solutions. Copyright © 2016 John Wiley & Sons, Ltd.