Center Manifold Analysis of Plateau Phenomena Caused by Degeneration of Three-Layer Perceptron

摘要

A hierarchical neural network usually has many singular regions in theparameter space due to the degeneration of hidden units. Here, we focuson a three-layer perceptron, which has one-dimensional singularregions comprising both attractive and repulsive parts. Such a singularregion is often called a Milnor-like attractor. It is empirically knownthat in the vicinity of aMilnor-like attractor, several parameters convergemuch faster than the rest and that the dynamics can be reduced to smallerdimensionalones. Here we give a rigorous proof for this phenomenonbased on a center manifold theory. As an application, we analyze the reduceddynamics near the Milnor-like attractor and study the stochasticeffects of the online learning.