A customizable MapReduce framework for complex data-intensive workflows on GPUs

摘要

The MapReduce programming model has been widely used in big data and cloud applications. Criticism on its inflexibility when being applied to complicated scientific applications recently emerges. Several techniques have been proposed to enhance its flexibility. However, some of them exert special requirements on applications, while others fail to support the increasingly popular coprocessors, such as Graphics Processing Unit (GPU). In this paper, we propose MR-Graph, a customizable and unified framework for GPU-based MapReduce, which aims to improve the flexibility and performance of MapReduce. MR-Graph addresses the limitations and restrictions of the traditional MapReduce execution paradigm. The three execution modes integrated in MR-Graph facilitates users to write their applications in a more flexible fashion by defining a Map and Reduce function call graph. MR-Graph efficiently explores the memory hierarchy in GPUs to reduce the data transfer overhead between execution stages and accommodate big data applications.We have implemented a prototype of MR-Graph and experimental results show the effectiveness of using MR-Graph for flexible and scalable GPU-based MapReduce computing.