MULTI-LAYERED NETWORKS WITH FREE ENERGY-BASED COMPETITIVE LEARNING

摘要

In this paper, we propose a new method to interpret and to improve the performance of multi-layered networks. In this method, we can select important competitive units by decreasing conditional entropy of competitive units for input patterns. As the entropy is decreased, only a small number of competitive units becomes activated, while all the other units are inactive. The conditional entropy is changed by decreasing the Gaussian width and we have no specific relevance measures to detect important competitive units. In addition, this conditional entropy is changed by using the free energy function similar to that in statistical mechanics. By using this free energy function, the heavy computation in conditional entropy is reduced to the computation of simple partition functions. We applied the method to the XOR problem and the cabinet approval ratings estimation. In the XOR problem, experimental results show that the number of hidden units effective in learning is reduced considerably by decreasing the Gaussian width σ. In addition, in the cabinet approval ratings estimation, experimental results confirmed that the number of important hidden units was considerably reduced by decreasing the Gaussian width, and generalization performance could be significantly improved.