Many scientific applications are I/O intensive and have tremendous I/O requirements, including checkpointing, periodically writing snapshots of computations. Especially, large number of these applications exhibits irregular access patterns, where accesses to data are performed through one or more level of indirections.; A typical computation science analysis cycle for these applications involves several steps: mesh generation, domain decomposition, simulation, visualization, archival of data, and adjustment of parameters. Therefore, two main focus must be considered. The first one is to store data set in a canonical form so that other steps can use it easily without having to reorganize. The second one is that, for a restart of computation with different number of processors, data set should be stored independent of number of processors that produced it.; In this dissertation, we present the design, implementation and evaluation of two parallel I/O runtime systems based on collective I/O techniques for irregular applications. The design is motivated by the requirements of a large number of science and engineering applications including teraflops, applications. The first library has been implemented on top of parallel file systems on MPPs. The user application links to the library's client API that issues I/O requests using the I/O commands supported by the parallel file systems. In this library, we designed and implemented two kinds of collective I/O schemes; "Collective I/O" and "Pipelined Collective I/O". In the "Collective I/O", all processors participate in the I/O simultaneously, and in the "Pipelined Collective I/O", I/O is overlapped with communication by making processor groups. As an optimization, chunking and on-line compression mechanisms are included in the both collective I/O schemes. The second library has been implemented on workstation clusters, called "Collective I/O Clustering". This library is based on the client-I/O server model. The I/O architecture of workstation clusters usually relies on a set of I/O servers, having local disks, and a set of diskless nodes. By using the local file system running at each node, such as UNIX, we developed a collective I/O scheme that supports irregular problems. In this library, two I/O configurations are possible. In the first configuration, all nodes have their local disk, thus the data in a client I/O buffer can go to its local disk, removing the communication between the clients and I/O servers. In the second configuration, only subset of nodes have their disk and can serve the incoming I/O requests from the clients. In this environment, the client sends the data to the appropriate I/O server, thus the communication latency between the client and I/O servers should be addressed to improve I/O performance. To optimize the communication latency, we provide a user-controllable stripe technique. Both the library and user applications can control this stripe unit. As did on MPPs, this library is incorporated with compression scheme to optimize I/O costs.; In this dissertation, the performance results on large-scale parallel systems including the Intel Paragon at Caltech, and ASCI/Red Teraflops at Sandia National Labs are presented. The results for the collective I/O clustering on IBM-SP located at Argonne National Labs are also presented.
展开▼
