Hyperplane 'Spin' Dynamics, Network Plasticity and Back-Propagation Learning

摘要

The processing performed by a feed-forward neural network is often interpretedthrough use of decision hyperplanes at each layer. The adaptation process, however, is normally explained using an error landscape picture. In the paper the actual dynamics of the decision hyperplanes is investigated. As a result a mechanical analogy is drawn with a system of spins acted upon by forces. The spin objects have a variable mass, and relaxation in the system is represented by increased overall system mass, also termed 'cooling'. The analogy is used to clarify the dynamics of learning, information storage and robustness, and in particular the functioning of the process of back-propagation. Learning deadlocks and local minima are explained. The concept of network 'plasticity' is introduced, and used to understand destructive relearning, and how it may thus be better avoided. Practical benefits from the interpretation are illustrated with hints for optimal weight initializations, avoidance of and escape from some types of local minima, and avoidance of destructive relearning through selective reordering of some types of networks. (Copyright (c) GMD 1992.)