A Principle Component Analysis Based Method to Normalize Term Weights

摘要

Term Weighting is a significant step in Document formalization in Natural Language Processing. It greatly interferes the accuracy of natural language processing systems. Term weight consists of three parts: Global Term Weight, Local Term Weight and standardization factor. Many term weight algorithms have been presented to address each part. And currently, the final term weight is the product of multiple term weight algorithms. However, the results of different term weight algorithms are correlated to each other, which indicates the redundant overlapped information between them. Simply multiplying the results leads to inaccurate final term weighting. This paper puts forward a Principle Component Analysis based Term Weights Normalizing Method, which is able to remove the redundant overlapped information and come up with a more accurate final term weight.