Operating two InfiniBand grid clusters over 28 km distance

摘要

This paper considers an InfiniBand connection between two bwGRiD clusters over a distance of 28 km in day-to-day production use. We discuss the hardware set-up of InfiniBand messages converted and transported over a fibre optic connection. The two clusters can be operated as single system image, the batch system will enforce that all nodes for a job are allocated on one side of the cluster. This is to optimise MPI performance, which would not be sufficient for communication between nodes on opposite sides of the 28 km connection. We report on the successful solution of all technical and organisational integration hurdles. By a simple performance model we discuss the anticipated costs for a doubling in communication performance.