IR+: Removing parallel I/O interference of MPI programs via data replication over heterogeneous storage devices

摘要

I/O requests from parallel processes to a disk compete with each other for a single disk head to access data. The disk efficiency can be significantly reduced due to frequent disk head seeks. In this paper, we propose a scheme, named IR+, to eliminate I/O interference for a single MPI program on disks by taking advantage of optimized access patterns and heterogeneous storage devices (e.g., solid-state disks). It identifies segments of files that could be involved in the interfering accesses and replicates them to their respectively designated storage servers. Moreover, it can further improve I/O performance by storing replicas on SSDs when accessing the on-disk replicas cannot sufficiently address the issue. IR+ has been implemented in the MPI library for high portability on top of the Lustre parallel file system. Our experiments with representative benchmarks, such as NPB BTIO and GTS, show that it can significantly improve I/O performance on the production HPC systems. (C) 2018 Elsevier B.V. All rights reserved.