Efficient design for MPI asynchronous progress without dedicated resources

摘要

The overlap of computation and communication is critical for good performance of many HPC applications. State-of-the-art designs for the asynchronous progress require specially designed hardware resources (advanced switches or network interface cards), dedicated processor cores or application modification (e.g. use of MPLTest). These techniques suffer from various issues like increasing code complexity/cost and loss of available compute resources for end applications. In this paper, we take up this challenge and propose a simple yet effective technique to achieve good overlap without needing any additional hardware or software resources. The proposed thread-based design allows MPI libraries to self-detect when asynchronous communication progress is needed and minimizes the number of context-switches and preemption between the main thread and the asynchronous progress thread. We evaluate the proposed design against state-of-the-art designs in other MPI libraries, including MVAPICH2, Intel MPI, and Open MR. We demonstrate the benefits of the proposed approach at microbenchmark level and at the application level at scale on five different architectures, including Intel Broadwell, Intel Xeon Phi (KNL), IBM OpenPOWER, and Intel Skylake with InfiniBand and Omni-Path interconnects. Compared to other state-of-the-art designs, our proposed approach shows up to 60%, 36%, and 50% improvements for All-to-one, One-to-all, and All-to-all communication patterns respectively at 816 processes. Our design shows 41% performance improvement for SPEC MPI compute-intensive applications, 63% performance improvement for P3DFFT application, and 28% higher throughput for HPL. Published by Elsevier B.V.