Learning Simpler Language Models with the Differential State Framework

摘要

Learning useful information across long time lags is a critical and difficultrnproblem for temporal neural models in tasks such as language modeling.rnExisting architectures that address the issue are often complex andrncostly to train. The differential state framework (DSF) is a simple andrnhigh-performing design that unifies previously introduced gated neuralrnmodels. DSF models maintain longer-term memory by learning to interpolaternbetween a fast-changing data-driven representation and a slowlyrnchanging, implicitly stable state.Within theDSF framework, a new architecturernis presented, the delta-RNN. This model requires hardly any morernparameters than a classical, simple recurrent network. In language modelingrnat the word and character levels, the delta-RNN outperforms popularrncomplex architectures, such as the long short-term memory (LSTM)rnand the gated recurrent unit (GRU), and, when regularized, performsrncomparably to several state-of-the-art baselines. At the subword level,rnthe delta-RNN’s performance is comparable to that of complex gatedrnarchitectures.