Optimization of Cross-Lingual LSI Training Data

摘要

The technique of latent semantic indexing (LSI) is widely employed in applications to provide information retrieval, categorization, clustering, and discovery capabilities. In these applications, the key relevant feature of the technique is the ability to compare objects (such as documents and queries) based on the semantics of their constituents. These comparisons are carried out in a high-dimensional vector space. That space is generated based on an analysis of occurrences of features in items of a training set. In the LSI literature there are multiple references to the fact that training items should be selected that are similar in content to the items to be dealt with in the application. This paper presents a principled approach for making such selection. We present test results for the technique for cross-lingual document similarity comparison. The results demonstrate that, at least for this use case, employment of the technique can have a dramatic beneficial effect on LSI performance.