Fusing Gini Index and Term Frequency for Text Feature Selection

摘要

Automatic text classification is the key technology to process and organize large-scale text data. It is well known that the high dimensionality of feature space is a main challenge for text classification. In order to attenuate such a problem as well as inspired by existing arts, we propose an effective text feature selection algorithm by novelly fusing the classical methodologies of Gini index and term frequency (TF), which is named as Gini-TF. Specifically, the involved Gini-TF function is wisely constructed by combining the Gini index text feature selection based on purity and the prior typical term frequency-inverse document frequency (TF-IDF) methods. Such a computation-efficient fusion would be beneficial for improving the efficacy of text feature selection. Experimental results show that our proposed Gini-TF fused algorithm could efficiently reduce the dimension of text feature space and improve the accuracy of text classification comparing with some prior classical methods.