Categorization of Multilingual Text on Languages of Indic Script

摘要

Due to the social media revolution in the world wide web of internet, a new aspect of language mixing and the ensuing language processing has opened up. There are a lot of messengers, applications and social websites that backup various languages for posting our messages and comments. Language detection and identification is a task of supervised machine learning which maps a text onto a unique language from among a set of trained languages. People many often use mixed languages in their communications. Identification of a native language from the mixed data is important as this conveys useful information. The present work explores Supervised Learning Method like K-Nearest Neighbor (KNN) technique, and character level n-gram method for the purpose of categorization of English, Bengali, Assamese, Marathi and Hindi web documents.