多源异构的智能配用电数据存储处理技术

摘要

针对智能配用电数据具有海量、多源异构等特点，提出一种基于 Hadoop 的智能配用电数据存储处理的框架设计方案。在对智能配用电数据组成进行梳理的基础上，提出了智能配用电的资源、存储和查询三层数据存储框架。其中，资源层采用 Hadoop 集群架构，结合电力系统的行业特色，完成IT存储资源管理；存储层则先采用XML技术进行数据预处理，使非结构化数据归一化，然后利用NoSQL数据库通过Map和Reduce的有效结合，完成海量数据的快速存储；查询层利用 To p-k 检索技术，实现海量数据的快速查寻。该设计方案能够对结构化和非结构化数据进行统一存储，可为智能配用电大数据技术的应用提供基础支撑。%A Hadoop-based management framework design is proposed for massive and multi-source isomerized data in smart power distribution and utilization. This framework consists of resource layer, storage layer and query layer, based on the data composition of smart power distribution and utilization. The resource layer uses Hadoop-cluster architecture. Combined with the characteristics of power system, the management of IT storage resource is completed. The storage layer firstly adopts XML technology framework to preprocess unstructured data aiming to normalize the isomerized data, and then the fast storage process of mass data can be done through the effective combination of Map and Reduce based on NoSQL. The query layer is able to fast achieve the searching of mass data through the implementation of Top-k technology. The proposed scheme can give a unified treatment to the storage and processing of structured and unstructured data, and provide a basic support for the application of big data technology for smart power distribution and utilization.