Convergence Study in Extended Kalman Filter-Based Training of Recurrent Neural Networks

摘要

Recurrent neural network (RNN) has emerged as a promising tool in modeling nonlinear dynamical systems, but the training convergence is still of concern. This paper aims to develop an effective extended Kalman filter-based RNN training approach with a controllable training convergence. The training convergence problem during extended Kalman filter-based RNN training has been proposed and studied by adapting two artificial training noise parameters: the covariance of measurement noise $(R)$ and the covariance of process noise $(Q)$ of Kalman filter. The $R$ and $Q$ adaption laws have been developed using the Lyapunov method and the maximum likelihood method, respectively. The effectiveness of the proposed adaption laws has been tested using a nonlinear dynamical benchmark system and further applied in cutting tool wear modeling. The results show that the $R$ adaption law can effectively avoid the divergence problem and ensure the training convergence, whereas the $Q$ adaption law helps improve the training convergence speed.