MODELING ARABIC LANGUAGE USING STATISTICAL METHODS

摘要

In this paper, we propose to investigate statistical language models for Arabic. First, several experiments using different smoothing techniques are carried out on a small corpus extracted from a daily newspaper. The sparseness of the data leads us to investigate other solutions without increasing the size of the corpus. A word segmentation technique has been employed in order to increase the statistical viability of the corpus. An n-morpheme model has been developed which leads to a better performance in terms of normalized perplexity. The second experiment concerns the study of the behavior of statistical models based on different kinds of corpora. The introduction of a distant n-gram improves the baseline model. Finally, we propose a comparative study of statistical language models for Arabic and several foreign languages. The objective of this study is to understand how to better model each of these languages. For foreign languages, trigram models are most appropriate whatever the smoothing technique used. For Arabic, the n-gram models of higher order smoothed with the Witten-Bell method are more efficient.