Turkish word n-gram analyzing algorithms for a large scale Turkish corpus - TurCo

摘要

To calculate some statistical properties of a language, first you need to take some samples of that language. That sample is called a corpus. An unbalanced large scale Turkish text corpus (TurCo) having /spl sim/362 MB capacity and more than 50 million words was prepared by using 12 different resources including Web sites and novels in Turkish language. Different algorithms were tested to obtain the n-gram (1/spl les/spl les/5) values. Efficiencies of different algorithms have been examined by applying them onto the each piece of the corpus one by one. Only detailed results of the two algorithms created without using database tables are given, because all the other algorithms need to run more than one day which makes those tests inefficient.