Learning to predict a context-free language: analysis of dynamics in recurrent hidden units

摘要

Recurrent neural network processing of regular languages is reasonably well understood. Recent work has examined the less familiar question of context-free languages. Previous results regarding the language a{sup}nb{sup}n suggest that while it ispossible for a small recurrent network to process context-free languages, learning them is difficult. This paper considers the reasons underlying this difficulty by considering the relationship between the dynamics of the network and weightspace. We areable to show that the dynamics required for the solution lie in a region of weightspace close to a bifurcation point where small changes in weights may result in radically different network behaviour. Furthermore, we show that the error gradientinformation in this region is highly irregular. We conclude that any gradient-based learning method will experience difficulty in learning the language due to the nature of the space, and that a more promising approach to improving learning performancemay be to make weight changes in a non-independent manner.