A Mixed Method Lemmatization Algorithm Using a Hierarchy of Linguistic Identities (HOLI)

摘要

We present a new mixed method lemmatizer for Icelandic, Lemmald, which achieves good performance by relying on IceTagger for tagging and The Icelandic Frequency Dictionary corpus for training. We combine the advantages of data-driven machine learning with linguistic insights to maximize performance. To achieve this, we make use of a novel approach: Hierarchy of Linguistic Identities (HOLI), which involves organizing features and feature structures for the machine learning based on linguistic knowledge. Accuracy of the lemmatization is further improved using an add-on which connects to the Database of Modern Icelandic Inflections. Given correct tagging, our system lemmatizes Icelandic text with an accuracy of 99.55%. We believe our method can be fruitfully adapted to other morphologically rich languages.