Towards a storage backend optimized for atomic MPI-I/O for parallel scientific applications

摘要

Scientific applications are becoming increasingly data-intensive: high-resolution simulations of natural phenomenas, climate modeling, large-scale image analysis, etc. Such applications currently manipulate data volumes in the petabyte scale and with the growing trend of data sizes we are rapidly advancing towards the exabyte scale. In this context, I/O performance has been repeatedly pointed out as a source of bottleneck that negatively impacts the performance of the applications. One of problems that is causing the I/O bottleneck are the I/O access patterns generated by such scientific applications that do not match the I/O access interfaces exposed by the file systems used as underlying storage back-ends. One particularly difficult challenge in this context is the need to efficiently address the I/O needs of scientific applications [1], [2] that partition multi-dimensional domains into overlapping subdomains. The subdomains need to be processed in parallel and then stored in a globally shared file. Since the file is a flat sequence of bytes, subdomains map to non-contiguous regions in the file. Because the subdomains overlap, under concurrent accesses such non-contiguous regions may interleave in an inconsistent fashion if they are not grouped together as a single atomic transaction. Therefore, atomicity of non-contiguous, overlapping reads and writes of data from a shared file is a crucial issue.