Statistical analysis of the effect of noise injection during neural network training

摘要

It is commonly accepted that the modification of the weights during training of an Artificial Neural Network can be augmented by addition of a random element chosen from various distributions. This technique, referred to as Noise Injection, allows the training process to stochastically traverse a larger subset of the sample space, as well as escape from local minima. This paper examines the effect of noise injection on the training cycle of feedforward neural networks. Emphasis is placed on the gradient descent weight modification technique of the backpropagation model. Statistical examination is made of the distribution of the effect within the topology of the weight space, upon the inputs to individual units, upon training time, and on the total error of the network. Since the weights of the network can be considered together as an n-tuple, injection of noise can be statistically examined within that n- dimensional space. It is shown that, for stochastically independent random distributions, the effect on this weight space and on the inputs to individual units is dependent upon the number of weights in the network. The multivariate distribution of the vector modification during training becomes increasingly distorted as the network size increases, such that noise injection has a more significant, and less stable, effect. Problems with traditional approaches are examined and an alternative noise injection method based on network size is presented.