identification of words in japanese text by a computer system

摘要

A word breaking facility operates to identify words within a Japanese text string. The word breaking facility performs morphological processing to identify postfix bound morphemes and prefix bound morphemes. The word breaking facility also performs opheme matching to identify likely stem characters. A scoring heuristic is applied to determine an optimal analysis that includes a postfix analysis, a stem analysis, and a prefix analysis. The morphological analyses are stored in an efficient compressed format to minimize the amount of memory they occupy and maximize the analysis speed. The morphological analyses of postfixes, stems, and prefixes is performed in a right-to-left fashion. The word breaking facility may be used in applications that demand identity of selection granularity, autosummarization applications, content indexing applications, and natural language processing applications.