An Implementation of Content-Based Pub/Sub System via Stream Computation

摘要

The sheer volume of data delivered via the Internet requires a more flexible and powerful communication model. As an expressive loosely-coupled asynchronous messaging model, Publish-Subscribe (Pub/Sub) system has been widely used. Traditional topic-based Pub/Sub system fails to understand the information of messages delivered, all messages must be previously classified into a set of topics. Content-based Pub/Sub system can dynamically choose subscribers for each message by its metadata. Existing distributed Pub/Sub systems are built on the overlay network consists of message brokers, which can adapt to heterogeneous network but inevitably impairs performance. In this paper, we designed a novel centralized tiered content-based Pub/Sub system with a four-layer architecture. In access layer, a customized naming strategy is proposed to achieve high availability. Internal message routing is finished in routing layer and sharding scheme is used to lower routing overhead. In computation layer, a two-step streaming computation model is used to boost the performance. In storage layer we adopt column-oriented database HBase for persistence. A set of comprehensive experiments were conduct to verify that our system achieve excellent performance, linear scalability and high availability.