Words Are Not Temporal Sequences of Characters

摘要

Language modeling is a valuable component of generative natural language processing (NLP) tasks, and benefits from explicit representations of the inherent hierarchies in language. We investigate a commonly used architecture that captures the concept that words are built from characters, and modify the word encoding mechanism to use a feed forward neural network rather than a recurrent neural network (RNN). This feed forward architecture facilitates increased performance and a reduction in the number of parameters over models that use common RNN implementations. We investigate whether word representations benelit from position- invariant features in the characters, and lind that fixed-position representations are sufficient.