Transparent Redirection of File-Based Data Accesses for Distributed Scientific Applications

摘要

File-based data access represents one of the most common ways of providing input data and retrieving output data in scientific applications. However, in distributed computing environments, the execution platform of the application and the storage location of the data might be different. Thus, additional efforts for performing the data transfers are required either by the application programmer or user. In this article, we propose an approach for accessing non-local data files with existing applications that can be performed by a regular user. The existing file-based data access operations of an application are intercepted and redirected to dedicated storage services with a specialized communication library. The approach is transparent in the sense that the application does not have to distinguish whether the data access is redirected or not. We present an implementation that supports various POSIX file I/O operations. Performance results with a benchmark application and the mesh generator Gmsh are shown to demonstrate the good performance and the low overhead of the proposed approach.