Statistical Machine Translation Approach for Lexical Normalization on Indonesian Text

摘要

Lexical normalization is an important task to be performed on noisy data, such as social media posts, before using the data for further analysis. We examine the potential of Statistical Machine Translation (SMT) for normalization of Indonesian text using the translation unit on both phrase and character levels. We also used an external corpus to generate additional language model data and pre-normalization rules to enhance the SMT system. The result shows the SMT systems on both phrase and character levels are outperforming various baseline in Word Error Rate (WER) score and Bilingual Understudy Evaluation (BLEU) score. This research also demonstrates the effect of using an external language model and applying pre-normalization rules can further enhance the effectiveness of SMT systems in normalizing Indonesian text.