Stochastic Word-Spacing System with Dynamic Increase of Word List

摘要

The main aim of this work is to implement a stochastic Korean word-spacing system which is equally robust for both inner data and external data. Word-spacing in Korean is influential in determining semantic and syntactic scope. In order to cope with various problems of word-spacing, this study (a) presents a simple stochastic word-spacing system with only two parameters using the odds favoring the inner-spacing of a given syllable bigram as well as relative word frequencies, and (b) endeavors to (ⅰ) remove noise from the training data and (ⅱ) diminish training data-dependency by dynamically creating a candidate word with longest-radix-selecting algorithm. The system thus becomes robust against unseen words and offers a similar performance for both inner data and external data: it obtained a 98.35% and a 96.59% precision in word-unit correction for the inner test data and the external test data, respectively.