Theoretical and numerical analysis of learning dynamics near singularity in multilayer perceptrons

The multilayer perceptron is one of the most widely used neural networks in applications, however, its learning behavior often becomes very slow, which is due to the singularities in the parameter space. In this paper, we analyze the learning dynamics near singularities in multilayer perceptrons by using traditional methods. We obtain the explicit expressions of the averaged learning equations which play a significant role in theoretical and numerical analysis. After obtaining the best approximation on overlap singularity, the stability of overlap singularity is analyzed. Then we take the numerical analysis on singular regions. Real averaged dynamics near the singularities are obtained in comparison with the theoretical learning trajectories near singularity. In the simulation we analyze the averaged learning dynamics, batch mode learning dynamics and on-line learning dynamics, respectively.