Evaluating the Scaling of Graph-Algorithms for Big Data Using GraphX

摘要

Graph processing has achieved a lot of attention in different big data scenarios. In this paper, we present the design, implementation, and experimental evaluation of graph processing algorithms in two different application areas. First, we use semi-clustering as an example of an algorithm typically used social network analysis. Then, we examine an algorithm for collaborative filtering as typically used in E-Commerce scenarios. For both algorithms, we make use of Apache GraphX as an existing distributed graph processing framework based on Apache Spark. As GraphX does not include these two algorithms, we describe how to implement them using a combination of GraphX and the underlying Spark Core. Based on our implementation, we perform experiments to test the scalability of both the algorithms and the GraphX processing framework. The experiments show that different kinds of graph algorithms can be supported within the Spark framework. Furthermore, we show that for our test data the algorithms scale almost linearly when properly designed.