Text Analysis Case Study: Determining Word Frequency based on Azerbaijan top 500 websites.

摘要

Word Frequency Distribution (WFD) is one the most important sub-areas of Natural Language Processing (NLP) and Computational Linguistic. The reliability and quality of WFD results are highly dependent on the size and quality of the corpora. In this paper describes the ongoing project with aim to build a corpus Azerbaijani text AzWebCorpus. Top 500 websites in Azerbaijan are used as a text source for corpus building. Most of essential tools including Web Crawler, Text Cleaner, Tokenizer have been developed and several opensource tools have been used. Moreover, AzWebCorpus compared to another corpus AzBookCorpus built on text taken from electronic books in terms of word frequency. Same approach that used in this paper is applicable for other languages.