A Corpus Preprocessing Method for Syllable-Level Tibetan Text Classification

摘要

Text classification is one of the most common and important tasks in the application field of natural language processing. With the rapid development of machine learning landscape, deep learning has become the mainstream approach for implementing text classification applications. However, deep learning has high requirements on the scale and quality of corpus, therefore, it is particularly important to build large-scale and high-quality corpus. In order to improve the quality of Tibetan text classification corpus, based on the analysis of the research status of corpus preprocessing, this paper proposes a syllable level Tibetan text classification corpus preprocessing model, and presents the core module of a text normalization algorithm which we refer as TC_ TCCNL. The proposed method lays a foundation for the construction of Tibetan text classification corpus.