Job co-allocation strategies for multiple high performance computing clusters

摘要

To more effectively use a network of high per-formance computing clusters, allocating multi-process jobsacross multiple connected clusters becomes an attractivepossibility. This allocation process entails dividing theprocesses of a job among several clusters, which we refer toas co-allocation. Co-allocation offers the possibility of moreefficient use of computer resources, reduced turn-aroundtime and computations using numbers of processes largerthan processes on any single cluster. In order to realize thesepossibilities, effective co-allocation, ultimately, depends onthe inter-cluster communication cost. In this paper, we intro-duce a scalable co-allocation strategy called the MaximumBandwidth Adjacent cluster Set (MBAS) strategy. The strat-egy makes use of two thresholds to control allocation: one tocontrol the limit on bandwidth on usable inter-cluster com-munication links and another to control how jobs are split.A simulator that can simulate the dynamic behavior of jobsrunning across multiple clusters was developed and used toexamine the performance of the MBAS co-allocation strat-egy. Our results indicate that by adjusting the thresholds forlink level control and chunk size control in splitting jobs, theMBAS co-allocation strategy can significantly improve bothuser satisfaction and system utilization.