Loop Query of Big Data with Low Transmission Cost

摘要

At present, big data environment in electric power field develops. Our platform has a poor efficiency of data analysis and the main bottleneck is disk I/O time of fetch operations. In order to address the problem, we use the characteristics that several tasks usually execute on the same data sets simultaneously, and propose a Loop Query method based on Fetch operations sharing (LQF). It lets all the operations on the data set to process each record circularly, which largely improves the efficiency of data analysis. We also present a Data Placement algorithm with low Transmission cost (DPT). It extends the queries to the cloud environment and reduces the result transmission cost of fetch operations. Last extensive experiments show that the proposed LQF algorithm performs 75.4% better than the benchmark solution when the number of tasks is large, and the DPT algorithm has on average 47.6% less execution time than random placement method.