Research on distributed data skew join algorithm based on VGFR model

摘要

The join operation is one of the core content of complex analysis tasks dealing with massive data. How to dealing with the data skew occurred in the join process effectively is a big challenge to us. But join algorithm based on the fragment and replicate in the current Shared-Nothing distributed architecture, which on the one hand, introduced a large data transmission overhead when a large amount of data is used to copy, the other hand, can not be adjusted dynamically according to the real-time load. In this paper, we propose and implement a grouping fragment and replicate join strategy based on virtual node (VGFR), aimed at robustness in terms of the size of both join sides. at the same time, the mapping relationship between virtual node and actual node is dynamically determined according to the system's real-time load, which can not only solve the problem of data skew effectively but also guarantee load balancing capability of the fine grained. Experimental results demonstrate that the optimizationThe join operation is one of the core content of complex analysis tasks dealing with massive data. How to dealing with the data skew occurred in the join process effectively is a big challenge to us. But join algorithm based on the fragment and replicate in the current Shared-Nothing distributed architecture, which on the one hand, introduced a large data transmission overhead when a large amount of data is used to copy, the other hand, can not be adjusted dynamically according to the real-time load. In this paper, we propose and implement a grouping fragment and replicate join strategy based on virtual node (VGFR), aimed at robustness in terms of the size of both join sides. at the same time, the mapping relationship between virtual node and actual node is dynamically determined according to the system's real-time load, which can not only solve the problem of data skew effectively but also guarantee load balancing capability of the fine grained. Experimental results demonstrate that the optimization technique can improve the performance of join operation effectively and has the adaptability. technique can improve the performance of join operation effectively and has the adaptability.