Highly scalable memory-efficient parallel LDA in a shared-nothing MPP database

摘要

Latent Dirichlet allocation (LDA) analysis on a dataset is performed on an MPP relational database by distributing subsets of said dataset to a plurality of segments of the MPP database, and performing LDA analysis in parallel on the respective subsets on the plurality of segments using Gibbs sampling. An object library on each segment provides executable objects of user defined functions that can be called by an SQL query when the query requires functionality provided by an object.