Using the Hermite Regression Formula to Design a Neural Architecture with Automatic Learning of the 'Hidden' Activation Functions

摘要

The value of the output function gradient of a neural network, calculated in the training Points, plays an essential role for its generalization capability. In this Paper a feed forward neural architecture (Net) that tan learn the activation function of its hidden units during the training Phase is presented. The automatic learning is obtained through the joint use of the Hermite regression formula and the CGD optimization algorithm with the Powell restart conditions. This technique leads to a smooth output function of Net in the nearby of the training Points, achieving an improvement of the generalization capability and the flexibility of the neural architecture. Experimental results, obtained comparing Net with traditional architectures with sigmoidal or sinusoidal activation functions, show that the former is very flexible and has good approximation and classification capabilities.