Enhancement of DTP Feature Selection Method for Text Categorization

摘要

This paper studies the structure of vectors obtained by using term selection methods in high-dimensional text collection, We found that the distance to transition point (DTP) method omits commonly occurring terms, which are poor discriminators between documents, but which convey important information about a collection. Experimental results obtained on the Reuters-21578 collection with the k-NN classifier show that feature selection by DTP combined with common terms outperforms slightly simple document frequency.