Hidden Markov-based LDA Internet Sensitive Information Text Filtering

摘要

We are in an era of rapid development of Internet information [1], generating billions of text every day. Under the test of such a “digital torrent”, how to ensure the ecological security and healthy development of the Internet has become a technical challenge. Rocchio[2] put forward a linear classifier which is a classification algorithm based on linear vector space model. With the development of hardware devices, the machine learning model has become the mainstream. The main training method is linear regression [3], K-Nearest Neighbor[4], Neural Network Model[5] and Support Vector Machine[6]. This paper raises a hidden Markov model based on feature keywords-themes. This is a statistically based approach. We use the Textrank algorithm [7] to extract feature words from a large number of data sets. Using the Apriori algorithm [8] to quantify the implicit relationship between feature words, we can generate a feature word confidence level matrix and establish a HMM-LDA correlation model. According to the text document which can produce an associated state matrix and a probability state transition matrix, we can confirm the conversion probability of the visible state chain. Thus we can filter and identify sensitive Internet information text.(Abstract)