多源异构海量数据实时处理平台研究与应用

摘要

目前许多业务部门以多种形式记录各种海量且异构多源的业务数据，在进行高端决策时，对数据的读取及多维计算分析等方面效率较低，特别是对具有“时空”特性的数据处理。针对这类数据的多源异构性、海量性、高度时空性等特点，结合数据实时性需求，设计并实现了“多源异构海量数据实时处理平台”。该平台从多途径实时获取、处理多种海量数据文件，并实时保存到数据库，方便其他各业务系统更加直接、实时地使用各种实时数据，最终实现各业务系统间的多源异构海量数据的实时共享，同时该平台对其他领域的复杂数据处理也具有很好的参考价值。%At present many business departments record the mass and heterogeneous multi-source business data in a variety of formats. When carrying out decision-making at high-end,it is lower in efficiency of data reading and multidimensional calculation,especially in regard to the processing of data with "time-space"characteristics.According to the characteristics of such kind of data with multi-source heterogeneity,mass and high spatiotemporal properties,and combining with the real-time data demand,we design and implement the“multi-source heterogeneous mass data real-time processing platform”.The platform timely acquires and processes a variety of mass data files from multiple channels,then saves to database in time.It is helpful for other business systems to use all kinds of real time data more timely and directly,and finally realise the real-time sharing of multi-source heterogeneous mass data between the business systems.At the same time, the platform has a very good reference value for the complex data processing in other areas.