A novel quasi-Newton-based optimization for neural network training incorporating Nesterov's accelerated gradient

摘要

This paper describes a novel quasi-Newton (QN) based accelerated technique for training of neural networks. Recently, Nesterov's accelerated gradient method has been utilized for the acceleration of the gradient-based training. In this paper the acceleration of the QN training algorithm is realized by the quadratic approximation of the error function incorporating the momentum term as Nesterov's method. It is shown that the proposed algorithm has a similar convergence property with the QN method. Neural network trainings for the function approximation and the microwave circuit modeling problems are presented to demonstrate the proposed algorithm. The method proposed here drastically improves the convergence speed of the conventional QN algorithm.