一种基于Hive日志分析的大数据存储优化方法

摘要

从2002年起，某金融机构积累了大量的银行卡支付日志数据。随着业务的不断增长，数据集高速增长，原生的 Hive 的查询性能已经不能令人满意。文章研究了大数据及存储的现状，提出了一种基于存储日志的分析的 Hive 存储格式优化方法，通过该优化方法从查询时间和存储空间利用率两方面使系统性能得到提升，提升了查询效率。论文对该金融机构原有 Hive 存储系统通过基于存储日志分析的 Hive 存储格式优化方法进行改进，通过实际数据，充分证明了该方法的可行性。%Since 2002, a financial institution has accumulated a large number of bank card payment data log .With the high-speed growth of business and data sets, the query performance of native Hive is not satisfactory. This paper studies the data and the present situation of the storage. This paper proposes an optimized data storage strategy based on Hive log anal-ysis. In that way, the optimized Hive will improve the efficiency, shorten the query time and reduce the data storage space greatly. At last, using the method above, this paper modified Hive warehouse of a domestic financial institution. After test-ing, there is significantly improvement in the optimized system in terms of a query speed and disk space utilization.