Scalable Natural Language Understanding: From Scratch, on the Fly

摘要

A new approach to language modelling is proposed, Deep Structure Learning, orders of magnitudes faster than the existing neural networks based language models due to effective sparse (symbolic) coding. Language model consists of a hierarchy of computational layers, extracting structural relations between increasingly abstract sets of symbols. The complexity of learning scales linearly with the size of the model, contrary to quadratic complexity scaling of backpropagation learning.