ActiveSpaces: Exploring dynamic code deployment for extreme scale data processing

摘要

Managing the large volumes of data produced by emerging scientific and engineering simulations running onrnleadership-class resources has become a critical challenge. The data have to be extracted off the computingrnnodes and transported to consumer nodes so that it can be processed, analyzed, visualized, archived, and sornon. Several recent research efforts have addressed data-related challenges at different levels. One attractivernapproach is to offload expensive input/output operations to a smaller set of dedicated computing nodesrnknown as a staging area. However, even using this approach, the data still have to be moved from the stagingrnarea to consumer nodes for processing, which continues to be a bottleneck. In this paper, we investigaternan alternate approach, namely moving the data-processing code to the staging area instead of moving therndata to the data-processing code. Specifically, we describe the ActiveSpaces framework, which provides (1)rnprogramming support for defining the data-processing routines to be downloaded to the staging area andrn(2) runtime mechanisms for transporting codes associated with these routines to the staging area, executingrnthe routines on the nodes that are part of the staging area, and returning the results. We also present anrnexperimental performance evaluation of ActiveSpaces using applications running on the Cray XT5 at OakrnRidge National Laboratory. Finally, we use a coupled fusion application workflow to explore the trade-offsrnbetween transporting data and transporting the code required for data processing during coupling, and werncharacterize sweet spots for each option.