Backpropagation through time with fixed memory size requirements

摘要

A generalization of the backpropagation through time (BPTT) algorithm is presented, which, under reasonable assumptions, can lead to fixed memory size requirements. The idea is to model BPTT as the storage of activations and errors in tapped delay lines, and then generalize the tap delay line to a gamma memory. Since the depth of the gamma filter is T=K//spl mu/, where K is the filter order and /spl mu/ a scalar that can be varied between 0/spl mu/1, it is possible to achieve depth T with a fixed filter order K. The accuracy of the gradient computation is tested in an example.