Regularization of differential-algebraic equations and recurrent training of neural networks

摘要

In this paper, a mathematical framework is provided for the analysis of local and global convergence properties of continuous-time recurrent training schemes for dynamical neural networks. This framework combines concepts coming from the theory of differential-algebraic equations (DAEs) with some notions related to non-critical linearly implicit differential equations. The continuous-time analogue of recurrent backpropagation can be seen as a semi-explicit DAE where the differential term is linearly implicit Several regularizations of this DAE lead to singularly perturbed systems for which local convergence results may be proved under mild conditions, while the linearly implicit nature of the resulting equations allows for a study of global convergence.