基于HBASE的时空大数据关联查询优化

摘要

随着数字采集和存储技术的快速发展,视频监测系统得到快速普及,以此带来了海量的监测视频数据.与文本数据不同的是,监测数据具有时空特征,如何在规模庞大且动态增长的数据量下进行高效的查询成为许多时空数据应用所关心的问题.针对云存储体系结构中监测视频大数据高效的时空联合查询需求,充分利用时空特征值和属性特征值在应用中的关联关系,以及HBase数据库在海量查询方面的优良性能,提出了基于HBase Bloomfilter的时空大数据多重过滤机制,创新性地利用视频文件特征值之间的依赖与关联关系来安排rowkey索引键.在此基础上设计出两种时空关联查询算法.最后通过实验证明了算法在时空大数据查询方面的可行性、灵活性和高效性,对其他大数据关联查询应用有较好的指导意义.%With the rapid development of digital acquisition and storage technology, video surveillance system has been rapidly popularized, which brings a lot of monitoring video data.Different from the text data, the monitoring data has the characteristics of time and space, and how to carry out efficient query under the large-scale and dynamic growth of data becomes a concern of many spatial and temporal data applications.To meet the requirement of space-time joint query for efficient monitoring of large video data in cloud storage architecture, in this paper, we make full use of the relationship between the temporal and spatial eigenvalues and the attribute eigenvalues in the application and the excellent performance of the HBase database in the massive query, propose HBase Bloomfilter`s multi-filtering mechanism of large data in space and time, and make use of the dependency and association between the eigenvalues of video files to arrange rowkey index keys in an innovative manner.On this basis, two kinds of spatiotemporal correlation query algorithms are designed.Finally, the feasibility, flexibility and efficiency of the algorithm are proved by experiments.It is useful for other large data association queries.