Study on the Application of Feature Selection for Big Text Data Using Expected Cross Entropy

摘要

In practice, microblog text data are sparse, high-dimensional and large and they are called microblog big text data in this paper. Microblog big text data are one of the important web data sources containing a wealth of user information. This paper studies feature selection for big text data in microblog. This study is meaningful because many features are no useful to text classification. And it is able to reduce dimensions so as to reduce data complexities. This paper introduces Expected Cross Entropy (ECE) to select effective features from microblog text big data and at the same time, variance score (VS), information gain (IG) and mutual information (MI) are also applied to feature selection for big text data in microblog. The four methods are compared using model classification accuracy. It is indicated in the experiments that ECE performs better than the other three methods in feature selection in microblog big text data.