Research and Implementation of an Aquaculture Monitoring System Based on Flink, MongoDB and Kafka

摘要

With the rapid advancement of intelligent agriculture technology, the application of IoT and sensors in aquaculture domain is becoming more and more widespread. Traditional relational database management systems cannot store the large scale and diversified sensor data flexibly and expansively. Moreover, the sensor stream data usually requires a processing operation with high throughput and low latency. Based on Flink, MongoDB and Kafka, we propose and implement an aquaculture monitoring system. Among them, Flink provides a high throughput, low latency processing platform for sensor data. Kafka, as a distributed publish-subscribe message system, acquires different sensor data and builds reliable pipelines for transmitting real-time data between application programs. MongoDB is suitable for storing diversified sensor data. As a highly reliable and high-performance column database, HBase is often used in sensor data storage schemes. Therefore, using real aquaculture dataset, the execution efficiency of some common operations between HBase and our solution are tested and compared. The experimental results show that the efficiency of our solution is much higher than that of HBase, which provided a feasible solution for the sensor data storage and processing of aquaculture.