ENABLING REWIRE-AWARE MAPREDUCE CLUSTER IN DISAGGREGATED SYSTEMS

摘要

MapReduce processing is carried out in a disaggregated compute environment comprising a set of resource pools that comprise a processor pool, and a memory pool. Upon receipt of a MapReduce job, a task scheduler allocates resources from the set of resource pools, the resources including one or more processors drawn from the processor pool, and one or more memory modules drawn from the memory pool. The task scheduler then schedules a set of tasks required by the MapReduce job. At least one particular task in the set is scheduled irrespective of a location of data required for the particular task. In association with a shuffle phase of the MapReduce job, and in connection with the particular task, at least one connection between a processor and at least one memory module is dynamically rewired based on the location of the data required for the particular task, thereby obviating network transfer of that data.