Adaptive Learning Algorithms to Incroporate Additional Functional Constraints into Neural Networks

摘要

In this paper, adaptive learnign algorithms to obtain better generalization performance are proposed. We specifically designed cost terms for the additional functinnality based on the first and second order derivatives defined at hiden layer. In the course of training these additional cost functions penalize the input-to-output mapping sensitivity and large curvatures contained in training data, repectively. A gradient-descent method results in hybrid learning rules to combine the error back-propagation, Hebbian ruels and the simple weight decay rules. However, additional computational requirements to the standard error back-propagation algorithm are almost negligible.