Language model adaptation in Tamil language using cross-lingual latent semantic analysis with document aligned corpora

摘要

Unlike English, Tamil does not have sufficient volume of text corpus to build a reliable language model. In this work, domain independent language model has been built with 500 Tamil documents. To improve the language model, adaptation with translation lexicons in Tamil generated from English using cross-lingual latent semantic analysis (CLSA) has been employed. Since Tamil is an agglutinative language, usage of surface word forms in CLSA will not yield better translation accuracy. Lexical gap between English and Tamil words has been reduced by the proposed partial morphological analysis in Tamil. This has improved the translation accuracy. Experiments have been conducted with direct and topic-specific model adaptations to improve the domain independent model. Significant improvements have been obtained in terms of perplexity and word error rate.