Improving Big Data Application Performance in Edge-Cloud Systems

摘要

Data analysis is widely used in all domains of the economy. While the amount of data to process grows, the time criteria and the resource consumption constraints get stricter. These phenomena call for advanced resource orchestration for the big data applications. The challenge is actually even greater at the advent of edge computing: orchestration of big data resources in a hybrid edge-cloud infrastructure is challenging. The difficulty stems from the fact that wide-area networking and all its well-known issues come into play and affect the performance of the application. In this paper we present the steps we made towards network-aware big data application design over such distributed systems. We propose a HDFS block placement algorithm for the network reliability problem we identify in geographically distributed topologies. The heuristic algorithm we propose provides better big data application performance compared to the default block placement method. We implement our solution in our simulation environment and show the improved quality of big data applications.