Research on Data Cleaning in Text Clustering

摘要

The more reasonable method of data cleaning has been proposed according to situation that data cleaning mistake away words which have distinguish capacity in text clustering pre-treatment presently. This method considers the situation of new field words happening. For the problem of rare word filtering, consider both the importance degree of the word in the whole text collection, namely word frequency, and the importance in the text in which it appears, namely weightings. So this method avoids dividing it into existed category in order to achieve the goal of filtering comparatively accurately which make result of text clustering more precise. Text clustering is made by means of C-means algorithm at last and verifying this method improves the accuracy of text clustering result.