An HDF5 MPI Virtual File Driver for Parallel In-situ Post-processing

摘要

With simulation codes becoming more powerful, using more and more resources, and producing larger and larger data, monitoring or post-processing simulation data in-situ has obvious advantages over the conventional approach of saving to - and reloading data from - the file system. The time it takes to write and then read the data from disk is a significant bottleneck for both the simulation and subsequent post-processing. In order to be able to post-process data as efficiently as possible with minimal disruption to the simulation itself, we have developed a parallel virtual file driver for the HDF5 library which acts as an MPI-IO virtual file layer, allowing the simulation to write in parallel to remotely located distributed shared memory instead of writing to disk.