基于流式计算框架的实时数据库分区系统

摘要

In order to realize the efficient processing of large scale and dynamic partition information in the big data environment,a real-time database partitioning system is proposed combining with the flow computing framework.This system copes with the large scale and dynamic workloads by using stream computing technologies in the big data environment.It designs a real-time data partitioning algorithm to realize automatic and immediate generation of data partitions.The system realizes the scalability and high-throughput adaption by using the horizontal scaling mechanism of streaming computing framework.The experimental results show that the system can realize efficient and real-time database partition in big data environment.It has higher partitioning quality and lower time than tranditional partitioning algorithm.%为实现大数据环境下大规模动态分区信息的高效处理,结合流式计算框架,提出一种实时数据库分区系统.采用大数据环境下的流式计算技术处理大规模动态的工作负载,设计实时数据分区算法实现数据分区的自动与即时生成,并利用流式计算框架的水平扩展机制提高系统扩展性和吞吐量.实验结果表明,该系统可在大数据环境下实现高效、实时的数据库分区,与传统分区算法相比,具有更高的分区质量和更少的分区时间.