Running Resilient MPI Applications on a Dynamic Group of Recommended Processes

摘要

HPC systems run applications that can take several hours to executeand have to deal with the occurrence of a potentially large numberof faults. Most of the existing fault-tolerance strategies for thesesystems assume crash faults that are permanent events easily detected. This is not the case in several real systems, in particular in shared clusters, in which even the load variation may cause performance problems that are virtually equivalent to faults. In this work we present a new model to deal with this problem in which processes execute tests among themselves in orderto determine whether the processors (or cores) on which they are runningare recommended or non-recommended. Processes classified asrecommended form a Dynamic Group of Recommended Processes (DGRP) that runs the application. The DGRP is formed only by processes that havenot been tested as non-recommended by all DGRP processes. A processnot in the DGRP that is continuously tested as recommended can rejoin theDGRP after a round of consensus executed by DGRP processes. Experimental results are presented obtained from a MPI-based implementation in which the HyperQuickSort parallel sorting algorithm reconfigures itself at run time to tolerate up to N -1 faults (in a system with N processes) while sorting up to 1 billion integers.