基于Hadoop的交互式大数据分析查询处理方法

摘要

An interactive processing method of analysis and query of big data based on Hadoop aims to analyze and query large data fast, whose important feature is the rapid query speed. The method is able to run on a cluster with thousands of nodes,suitable for analyzing semi-structured or nested data,combining with existing SQL environment and Apache Hive. The main purpose is to use the method to connect HDFS,Hive and Hbase for query,also achieving to query data from different data sources. Furthermore,in the same Hadoop clus-tering environment,the method and Spark SQL is compared in the query speed for data with 100 000,200 000,500 000,one million and five million. Several experiments show the method is fast and efficient,and enables business users to query data and analyze enterprise Ha-doop big data quickly and efficiently.%基于Hadoop的交互式大数据分析查询处理方法旨在快速分析查询大数据集的信息，最重要的特征就是查询速度快。该方法能够运行在上千节点的集群上，适于半结构化/嵌套数据的分析、兼容现有的SQL环境和Apache Hive。文中主要利用此方法实现连接HDFS、Hive以及Hbase进行查询测试，还完成了同时从不同数据源上关联查询数据。在同一Ha-doop集群环境中，将该方法与Spark SQL对于10万、20万、50万、100万、500万条数据进行查询速度对比测试。经过多次实验后得出，基于Hadoop的交互式大数据分析查询处理方法速度快、效率高，能够帮助企业用户快速、高效地进行Hadoop数据查询和企业级大数据分析。