RECURRENT NEURAL NETWORK TRAINING WITH THE KALMAN FILTER-BASED TECHNIQUES

摘要

Recurrent neural networks, in contrast to the classical feedforward neural networks, handle better the inputs that have space-time structure, e.g. symbolic time series. Since the classic gradient methods for recurrent neural network training on longer input sequences converge very poorly and slowly, alternative approaches are needed. We describe the training method with the Extended Kalman Filter and with its modifications Unscented Kalman Filter, nprKF and with their joint versions. The Joint Unscented Kalman Filter was not used for this purpose before. We compare the performance of these filters and of the classic Truncated Backpropagation Through Time (BPTT(h)) in two experiments for next-symbol prediction - word sequence generated by Reber automaton and the sequence generated by quantising the activations of a laser in a chaotic regime. All the new filters achieved significantly better results.