In‐place query driven big data platform: Applications to post processing of environmental monitoring

摘要

This paper describes the use of an experimental big data platform for applications of environmentalrnmonitoring associated with visualization of global climate forecast data and air quality modelrnsimulation and response. Environmental monitoring in general requires both capabilities of modelrnsimulation for forecast, and data processing for visualization and analyses. The in‐place queryrndriven big data platform, based on concepts of Query Driven Visualization and shared‐nothingrndistributed database, thus is developed for the need. The system architecture of this experimentalrnbig data platform entails one master data node and 17 slave data nodes, while the system links tornthe National Center for High‐performance Computing supercomputer, Advanced Large‐scale ParallelrnSupercluster, and storage pool. For software implementation, the openSUSE operating systemrnand MariaDB database are installed on all nodes. The master data node is responsible forrnmetadata management and information integration and the 17 slave data nodes for distributedrndatabase and parallel model simulation, data visualization, and analyses. The application of globalrnclimate data visualization (Outgoing Longwave Radiation or OLR, temperature, rainfall, etc.) in thernplatform serves first to partition Network Common Data Form file data into shared‐nothing distributedrndatabases for partial visualization in slave data nodes, then integrated into whole visualizationrnin the master node through Message Passing Interface communication.rnFor the application of air quality management, we first accessed Taiwan Environmental ProtectionrnAdministration (EPA) observed data in the master node. EPA observed data are replicatedrnto distributed databases in slave nodes; and the air pollution model, Gaussian plume trajectoryrnmodel, is replicated in all slave nodes for model simulation, which produces output data and associatedrnimage files in the local file system. The master node is able to collect whole image filesrnthrough the remote shared file system for display of the results. We can see the approach of datarnI/O access in 2 applications, due to individual problem features, each application is unique. Examplesrnof benchmark cases reveal strong performance in accelerating computing speed and reducingrnthe I/O operational time. It is found that the platform is able to accelerate climate data visualizationrnprocesses, help research scientists gain the deep insights into data, and explore the potentialrnphenomena and features, such as formation of Typhoon eddies. In air quality management applications,rnthe platform is used to perform the air pollution model Gaussian plume trajectory model.rnBackward trajectory simulation of PM2.5 concentrations is used to identify the 30+ pointrnsource's contribution on 73 EPA monitor stations (receptors) in Taiwan. A user‐friendly, web‐servicernbased big data presentation uses the heterogeneous observed and forecast pollutant data inrnspace and time. The results support for air quality decision‐making and emergency response. Thernlimitation of data size for applications in the platform, the current users and future developmentrnof the platform, and the linkage of PRAGMA collaboration are also described in the paper.