A method of optimizing LDA result purity based on semantic similarity

摘要

The result purity of traditional LDA (Latent Dirichlet Allocation) is uninterpretable because it is always difficult to summarize the meaning of each LDA result topic which contains multiple irrelevant words. To solve the problem, a method of optimizing LDA result purity based on semantic similarity in streaming news processing is proposed. In this method, the Category Cluster Density (CCD) of each topic is calculated first, and those topics with lower CCD value were dropped to optimize the overall LDA result purity. The news clustering experiment results show that the vague news can be removed effectively and the reserved topics are interpretable than traditional method, which can significant optimize the LDA result purity automatically.