Hadoop海量数据迁移系统开发及应用

摘要

With more and more data generated by High Energy Physics(HEP)experiments, Hadoop has been a solution for HEP data analysis while facing with the demand of data migration. However, existing data migration tools do not sup-port data transmission between HDFS and other file systems, and have obvious performance deficiency. Based on the requirements of high-energy physical data synchronization and archiving, this paper designs and implements a universal mass data migration system, which uses MapReduce to directly move data between HDFS and other storage systems or media by extending the HDFS data access methods. In addition, dynamic priority scheduling model is proposed to do multi-tasks dynamic priority assignment and selection. The system has been applied to the data migration in LHAASO experiment, and the actual operation results indicate that the system achieves good performance and meets the data migra-tion requirements of various experiments.%当前高能物理实验产生的数据量越来越大,利用大数据处理平台Hadoop进行高能物理数据处理时,面临数据迁移的实际需求,而现有迁移工具不支持HDFS与其他文件系统间的数据传输,性能存在明显缺陷.从高能物理数据同步、归档等需求出发,设计和实现了一个通用的海量数据迁移系统,通过扩展HDFS数据访问方式,使用Map-Reduce直接在HDFS数据节点和其他存储系统/介质之间迁移数据.此外,系统设计实现了动态优先级调度模型,进行多任务的动态优先级评定和选取.该系统已经应用于大型高海拔空气簇射观测站(LHAASO)宇宙线等物理实验中的数据迁移,实际运行结果表明系统性能良好,能够满足各个实验的数据迁移需求.