Correlation between weighted sums in multi-layer perceptrons decreases under sigmoidal transformations

摘要

Nonlinear transformation is one of the major obstacles to analyzing the properties of multilayer perceptrons. In this paper, we prove that the correlation coefficient between two jointly Gaussian random variables decreases when each of them is transformed under continuous nonlinear transformations, which can be approximated to piecewise linear functions. When the inputs or the weights of a multilayer perceptron are perturbed randomly, the weighted sums to the hidden neurons are asymptotically jointly Gaussian random variables. Since the sigmoidal transformation can be approximated piecewise linearly, the correlations among the weighted sums decrease under the sigmoidal transformations. Based on this result, we can say that the sigmoidal transformation as the transfer function of the multilayer perceptron reduce the redundancy in the information contents of the hidden neurons.