Introduction and Correction of Bengali-Hindi Noise in Large Word Vocabulary using RNN

摘要

Word correction or Spell Checking is essential for fundamental applications such as text editors and social media based chat platforms. A lot of works have been reported in English, Chinese, and other European languages with available standard datasets. However, there is a lack of sizeable standard datasets with Indic languages, which is one of the causes not to observe enough contribution in this field. As the possibility of error in a word for an Indic script is different than that of English due to the presence of diacritics(matra), the synthetic noise introduction should be very specific. In the work, we focus on creating two large standard noisy datasets using the probabilistic rule on alphabets for Hindi and Bengali languages. Each dataset contains a vocabulary size of 150000 words. We tested the datasets using two recurrent neural networks (RNN) based models which provide 80% accuracy on both the datasets.