Automatic Discovery of Non-Compositional Compounds in Parallel Data

摘要

Automatic segmentation of text into minimal content-bearing units is anunsolved problem even for languages like English. Spaces between words offer aneasy first approximation, but this approximation is not good enough for machinetranslation (MT), where many word sequences are not translated word-for-word.This paper presents an efficient automatic method for discovering sequences ofwords that are translated as a unit. The method proceeds by comparing pairs ofstatistical translation models induced from parallel texts in two languages. Itcan discover hundreds of non-compositional compounds on each iteration, andconstructs longer compounds out of shorter ones. Objective evaluation on asimple machine translation task has shown the method's potential to improve thequality of MT output. The method makes few assumptions about the data, so itcan be applied to parallel data other than parallel texts, such as wordspellings and pronunciations.