Fast Oov Words Incorporation Using Structured Word Embeddings for Neural Network Language Model

摘要

Recently, deep learning approaches have been widely used in language modeling and achieved great success. However, the out-of-vocabulary (OOV) words are often estimated in a rather crude way using only one special symbol, which ignores the linguistic information. In this paper we present an LSTM language model with structured word embeddings to tackle this problem. In our model, both input and output embeddings of LSTM language model are deployed with structured word embeddings. Utilizing syntactic-level and morphological-level parameters sharing, OOV words can be incorporated into the proposed model without retraining. The LSTM language model with structured word embeddings is instantiated for Chinese. Experiments show that the proposed model achieves PPL improvement on OOV words, and can be further integrated into automatic speech recognition systems for fast vocabulary updating.