基于开源技术的网络数据采集系统设计

摘要

Based on analyzing the characteristics of network data col ection,it presents the design goal of network data col ecting sys-tem,that is supporting the real-time computing and querying of the key network index,supporting multi-data source and multi-consumer,supporting real-time col ection and batch col ection,and also has the linear extend capability. The system ar-chitecture is designed with the open source technologies,such as Flume,Kafka,Storm and Hadoop. The countermeasures are presented for the chal enge before architecture deployment.%通过分析网络数据采集的特点，提出了网络数据采集系统的设计目标，即支持关键网络指标实时计算和查询、支持多数据源和多消费者、支持实时采集和批量采集且具备线性扩展能力。采用Flume、Kafka、Storm、Hadoop等开源技术完成了系统架构设计。对架构实施可能面临的挑战提出了应对策略。