Non-parametric Discovery of Topics and Communities in Distributed and Streaming Environments

摘要

Several recent works have focused on improving latent-space based modeling of streaming count-based data such as streaming textual feeds and evolving social networks. However, many of these models do not inherently scale to large data sets, nor do they accommodate drift in the inferred latent factors (e.g. topics, social groups) over time. In addition, the functional form of distributed and streaming processing architectures recently introduced in industry places constraints on how dynamic algorithms can be expressed, for example, that they must be inherently state-ful. We propose a comprehensive and flexible approach to distributed and dynamic inference of Bayesian count factorization models, focusing on a recently introduced nonparametric, joint topic-community factorization model called Joint Gamma Process Poisson Factorization (JGPPF). The method is illustrated in an Apache Spark implementation using twelve years of U.S. Senate voting records.