A data collecting and processing method in high-speed network based on cluster

摘要

Through in-depth study cluster and network data processing rules and methods, a data collecting and processing method in high-speed network based on cluster is proposed in this paper. It realized network data distributed collecting by using multilayer structure of task scheduling centre, cluster management center, collecting agency. It realized network data processing by adopting data processing method based on the rules. In addition, It introduces data processing rules, cluster distribution strategy and management center choice mechanism in detail. The results of simulation experiment show that the method is suitable for large-scale network data collecting and processing, and can improve data collecting and processing speed very well.