Word sense induction in bengali using parallel corpora and distributional semantics

摘要

One of the most challenging research problems in natural language processing (NLP) is that of word sense induction (WSI). It involves discovering senses of a word given its contexts of usage without the use of a sense inventory which differentiates it from traditional word sense disambiguation (WSD). This paper reports a work on sense induction in Bengali, a less-resourced language, based on distributional semantics and translation based context vectors learned from parallel corpora to improve the task performance. The performance of the proposed method of sense induction was compared with the k-means algorithm, which was considered as the baseline in our work. A dataset for sense induction was created for 15 Bengali words, encompassing a total of 111 contexts. The proposed model, in both mono and cross-lingual settings, outperformed k-means in precision (P), recall (R) and F-scores. K-means based sense induction produced average P, R and F-scores of 0.71, 0.73 and 0.66 respectively. The average P,R and F-scores produced by the mono- and cross-lingual settings of the proposed algorithm are 0.77, 0.73, 0.68 and 0.81, 0.77 and 0.72 respectively.