可扩展的分布式元数据管理系统设计

摘要

To solve the problems caused by the storage expanding in high energy physics mass storage system, a scalable distributed metadata management system is designed, which includes metadata management, metadata service, cache service and monitoring information collector. Based on it, a new Adaptive Directory Sub-tree Partition ( ADSP ) algorithm is proposed. ADSP divides the file system namespace into sub-trees with directory granularity and adjusts sub-trees adaptively according to the load of metadata cluster for achieving the storage and distribution of metadata in cluster. Experimental results show that the algorithm can improve the metadata access and retrieval performance, provides a scalable and dynamic load balancing of metadata service to ensure the availability, scalability and I/O performance of metadata management system is not affected by the storage scale,thereby it can meet the growing storage requirements of high energy physics experiments.%为解决高能物理海量存储系统由于存储规模不断扩大所面临的问题，设计一种分布式元数据管理系统，包括元数据管理、元数据服务、缓存服务以及监控信息采集4个部分，在此基础上提出自适应目录子树划分算法，以目录为粒度进行元数据划分，根据集群负载情况调整目录子树，实现元数据信息在元数据集群中的合理存储和分布。实验结果证明，该算法能提高元数据的访问和检索性能，提供可扩展及动态负载均衡的元数据服务，以保证该元数据管理系统的可用性、扩展性及I/O性能不会因存储规模扩大而受到影响，满足高能物理实验日益增长的存储需求。