Research on Parallel Computing Method of Hydrological Data Cube

摘要

Hydrological data is characterized by large data volume and high dimension. In view of non-hierarchical dimension data in hydrological data, this paper proposed Segmented Parallel Dwarf Cube (SPD-Cube). Firstly, a high-dimensional cube is divided into several low-dimensional segments. Then by calculating the sub-dwarf of each low-dimensional segment, the storage space required for cube materialization is greatly reduced. Finally, combining Map/Reduce technology to construct, store and query several sub-dwarfs in parallel. Experiments show that SPD-Cube not only realizes highly compressed storage of data, but also realizes efficient query by combining Map/Reduce distributed parallel architecture.