Non-local Data Fetch Scheme Based on Delay Distribution for Hadoop Clusters in Public Cloud

摘要

Hadoop and its ecosystem have become the de facto platform for processing large-scale data, also known as Big Data, because it hides the complexity of distributed computing, scheduling, and communication while providing fault-tolerance. Most of the Hadoop's features are designed for on-premise data center hosted clusters where cluster topology is known. With cloud-based computing becoming more popular and mature, more and more users deploy Hadoop clusters in public cloud environments. Hadoop depends on administrator configured rack assignment of servers to calculate the distance between servers. When fetching non-local data, Hadoop calculates the distance between servers to find the best remote server to fetch data from. However, in public cloud environments, it is impossible to know the rack assignment of virtual servers leaving Hadoop to fetch data from a remote server that is on the other side of the data center sometimes. To overcome this problem, we propose a delay distribution based scheme to find the closest server to fetch data from. The proposed scheme selects a server comparing the delay distributions between server pairs. Delay distribution is calculated measuring the round-trip time between servers periodically. Our experiments observe that the proposed scheme outperforms conventional Hadoop nearly by 12% in terms of non-local data fetch time. This reduction in data fetch time will lead to the reduction in job runtime, especially in real-world multi-user clusters where non-local data fetching can happen frequently.