A Data-Aware Scheduling Framework for Parallel Applications in a Cloud Environment

摘要

Cloud infrastructures are competent to providing massive processing capabilities of computational and data resources in virtualized environments. Introduction of big data analytics in many spheres of science, technology and business has led to the trend of employing data-parallel frameworks, like Hadoop for handling such massive data requirements. Since most Hadoop based systems make the two decisions of scheduling data and computation independently, it seems a promising prospective to map computations within cloud resources based on data blocks already distributed to them. This paper proposes a computation scheduling framework that adopts the strategy of improving computation and data co-allocation within a Hadoop cloud infrastructure based on knowledge of data blocks availability, hereafter referred to as Data Aware Scheduling (DAS) framework. The proposed DAS employs a dependency based grouping of data. Experiments have been conducted using standard map-reduce applications and results presented herein conclusively demonstrate the efficacy of the proposed framework.