基于 HDFS 的创新知识云平台存储架构的研究与设计

摘要

Because current storage structure cannot meet the storage and service demands brought by massive innovative knowledge,we presented an improved HDFS distributed storage system and applied it to the innovative knowledge cloud platform.First,we introduced the package file and distributed indexing service,thus improved the efficiency of HDFS small files storage problem.Then by optimising the naming space backup and failure recovery services in HDFS we achieved high availability architecture of HDFS with stronger availability and higher resource utilisation.By designing and implementation of the system we proved that the optimisation work greatly reduced the memory pressure of naming node and enhanced the availability of cluster,and the improved HDFS storage system could meet the storage needs of innovative knowledge cloud platform.%针对现有存储结构无法满足海量创新知识带来的存储及服务需求的问题，提出一种改进的 HDFS（Hadoop Distributed File System）分布式存储系统并应用到创新知识云平台。首先引入包文件及分布式索引服务，改进 HDFS 小文件存储的效率问题，然后通过优化 HDFS 的命名空间备份及故障恢复服务，实现可用性更强、资源利用率更高的 HDFS 高可用架构。通过系统的设计和实现证明优化工作大大降低了命名节点的内存压力，提高了集群的可用性，并且改进的 HDFS 存储系统可以满足创新知识云平台的存储需求。