Kernel Hidden Unit Analysis: Network Size Reduction by Entropy Minimization

摘要

In this paper, we propose a method, called Kernel Hidden Unit Analysis, to reduce the network size. The kernel hidden unit analysis is composed of two principal components: T-component and S-component. The T-component transforms original networks into the networks which can easily be simplified. The S-component is used to select kernel units in the networks and construct kernel networks with kernel units. For the T-component, an entropy function is used, which is defined with respect to the state of the hidden units. In a process of entropy minimization, multiple strongly inhibitory connections are to be generated, which tend to turn off as many units as possible. Thus, some major hidden units can easily be extracted. Concerning the S-component, we use the relevance and the variance of input-hidden connections and detect the kernel hidden units for constructing the kernel networks. Applying the kernel hidden unit analysis to the symmetry problem and autoencoders, we perfectly succeeded in obtaining kernel networks with small entropy, that is, small number of hidden units.