A NOVEL MASS METEOROLOGICAL DATASTORAGE SYSTEM BASED ON HADOOP ECOSYSTEM

摘要

When using the original system to store massive meteorological data,when the data volume is in the range of 100.25 TB-130.25 TB,it usually has the problem of slow data transmission rate.Therefore,in this paper,we propose a massive meteorological data storage system based on the Hadoop ecosystem.The hardware of the system includes a server module and a weather data storage module.Among them,the server module is composed of a database remote server,an application server,etc.; the meteorological data storage module is composed of a Vault Power TM module,a main server,etc.The software of the system includes data visualization module,data processing module and database module.The data visualization module can realize the data visualization processing of meteorological point elements and meteorological linear elements.The data processing module can realize scalable,efficient and reliable distributed data processing of massive meteorological data.While the database module is composed of system management tables,file library management and meteorological data tables.The combination of hardware and software effectively realizes the storage of massive meteorological data.Finally,through comparative experiments and verification,from the perspective of ecological environment protection,the data transmission rate of the system is higher than the original system in the range of 100.25TB-130.25TB,and the data transmission performance is improved.