Removing potential flat spots on error surface of multilayer perceptron (MLP) neural networks

摘要

As the derivative of the sigmoid activation function approaches zero, the back propagation algorithm involves flat spots on the error surface of multilayer perceptron (MLP) neural networks, which means the hidden neurons of MLP were saturated. Flat spots can slow down the gradient search and hamper convergence. In this paper, we propose a grading technique to gradually level off the potential flat spots to a sloping surface in a look-ahead mode; and thereby progressively renew the saturated hidden neurons. We introduce a criterion to measure the saturation level of MLP, and then we modify the error function by using a proposed piecewise error function that switches between two cases, regarding the level of MLP saturation. These two cases include the standard error function, when MLP is not saturated, and the modified error function, when MLP is saturated. We recorded considerable improvements, especially in convergence rate and generalization, on the tested benchmark problems.