Resilient distributed computing platforms for big data analysis using Spark and Hadoop

摘要

This paper introduces the integration of three platforms using Apache Hive, Cloudera Impala and BDAS Spark SQL which enables to support SQL-like queries in big data environment. In order to fast respond to user's query for big data processing, the optimized system can automatically select the appropriate platform to best perform a query. In addition, the rapid data retrieval from the in-memory cache or in-disk cache has achieved for the repeated SQL command. The proposed approach improves the efficiency of data retrieval significantly.