Levenberg-Marquardt Learning and Regularization

摘要

Levenberg-Marquardt Learning was first introduced to the feedforward net-works to improve the speed of the training. This method is an improved Guass-Newton method which has an extra term to prevent the cases if ill-conditions. Interestingly, If we regard the learning as a constrianed least square method, that extra term becomes a regularization term to deal with the additive noise in the training samples. In this paper, we look at the Levenbrg-Marquardt Learning from the viewpoint of regularization. We show that the Levenberg-Marquardt learning allows other forms of regularization operators by some simple modifications. In addition, with the inclusion of test for validation error, the regularization parameter can be chosen in such a way that both the training error and validation error decrease. Thus, it prevents the occurrence of over-training.