Inverse Clustering-Based Job Placement Method for Efficient Big Data Analysis

摘要

To efficiently exploit the inherent values of big data, the large-scale data center with multiple compute nodes is deployed. In this scenario, the job placement method becomes the key issue to match the compute nodes with the data analysis jobs, to balance the workloads among the nodes and meet the resource requirements for various jobs. In this work, an inverse clustering-based job placement method is proposed. Jobs are represented as feature vectors of resource utilizations and priorities. Then contrary to the regular clustering procedure, the proposed inverse clustering method organizes jobs with the largest different feature vectors into the same groups. Jobs in the same groups are placed on to the same nodes. Consequently, jobs assigned on the same nodes utilize different types of resources and are labeled with different priorities. In our simulation experiments, a global load and priority balances are achieved with the proposed inverse clustering method.