A flexible I/Oarbitration framework for netCDF-based big data processingworkflows on high-end supercomputers

摘要

Onthe verge of the convergence between high-performance computing and BigData processing,rnit has become increasingly prevalent to deploy large-scale data analytics workloads on high-endrnsupercomputers. Such applications often come in the form of complex workflows with variousrndifferent components, assimilating data from scientific simulations as well as from measurementsrnstreamed from sensor networks, such as radars and satellites. For example, as part ofrnthe Flagship 2020 (post-K) supercomputer project of Japan, RIKEN is investigating the feasibilityrnof a highly accurate weather forecasting system that would provide a real-time outlookrnfor severe guerrilla rainstorms. One of the main performance bottlenecks of this application isrnthe lack of efficient communication among workflow components, which currently takes placernover the parallel file system.In this paper, we present an initial study of a direct communicationrnframework designed for complex workflows that eliminates unnecessary file I/O among components.rnSpecifically, we propose an I/O arbitration layer that provides direct parallel data transferrn(both synchronous and asynchronous) among job components that rely on the netCDF interfacernfor performing I/O operations. Our solution requires only minimal modifications to applicationrncode.Moreover, we propose a configuration file–based approach that allows users to specify therndesired data transfer pattern among workflow components, offering a general solution for differentrnapplication contexts.We present a preliminary evaluation of the proposed framework onrnthe K Computer (running on up to 4800 compute nodes) using RIKEN's experimental weatherrnforecastingworkflow as a case study.