Dynamic Load Balancing and Channel Strategy for Apache Flume Collecting Real-Time Data Stream

摘要

With the development of information technology, real-time data stream processing(RTDSP) has become a popular research topic. The first step of RTDSP is collecting data, requiring a data collector to receive data from the source and send them to the sink. Apache Flume, a distributed and reliable framework, used for this purpose, has some limitations and drawbacks on load balancing and storage. In this paper, we aim to improve performance and availability for collecting unstable real-time big data stream. So we propose a new load balancing strategy based on the free memory size and a storage strategy of integration memory channel with the multi-file channel to reduce the overhead of disk and network. Finally, the experimental results show that the availability and performance are improved under the condition of a poor network, high availability requirements, intense competition in memory resources and large data size. Specifically, the availability is higher than 99.999%, and the performance can be improved by 10%-50% under different conditions.