Research on Mass News Classification Algorithm Based on Spark

摘要

In recent years, with the explosion of the number of Internet news, people pay more and more attention to how to classify the mass of news. Therefore, this paper studies the mass news classification algorithm based on Spark, aiming at the problem of how to classify mass news data quickly and efficiently. In this paper, a large amount of news text is segmented based on Jieba segmentation tool, and several versions of stop words list are combined to remove stop words. Secondly, on the basis of traditional convolutional neural network, this paper proposes a news classification algorithm based on the combination of pre-trained Word2vec and improved CNN. In addition, the classification algorithm proposed in this paper is parallelized based on Spark, which improves the speed of mass news classification. In this paper, the standard data sets are used to compare and experiment the proposed news classification algorithm. The experimental results show that compared with the traditional algorithm, the news classification optimization algorithm designed in this paper has obvious improvement in multiple evaluation indexes such as accuracy, recall and F1. In addition, after parallel design of the algorithm proposed in this paper based on Spark, compared with the serial algorithm, the speed improvement effect is also more significant.