Hobbits: Hadoop and Hive based Internet traffic analysis

摘要

Internet traffic measurement and analysis have long been used to characterize network usage and user behaviors, but face the problem of scalability under the explosive growth of Internet traffic and high-speed access. In this paper, we present Hobbits, a Hadoop and Hive based traffic analysis system that performs Internet Protocol (IP) and Transport Control Protocol (TCP) analysis of large-sized Internet traffic in a scalable manner. Our experimental evaluation on real datasets confirms that Hobbits outperforms previous solutions in terms of both job completion time and storage efficiency.