Utilizing Previous Weight Estimates for Accelerated Training

摘要

It is easily seen that if the error function is quadratic anda linear combination of several previos gradients is the zero vector, then the same linear combination of the weights at those points yields the minimizing weight. The former condition can be made highly likely by considering one more gradient than the dimension of the weight vector. From this simple observation, we first generalize to the case of linear combinations of fewer gradients than the dimension of the weight vector, and then to a general algorithm pplicable to non-quadratic error functions which permits us to calculate the coefficients of the linear combination and control their number in a computationally inexpensive manner. A hybrid of this algorithm iwht the traditional backpropagation algorithm is seen to yield substantial reduction in computation time over the backpropagation algorithm alone.