Boundedness and convergence of batch back-propagation algorithm with penalty for feedforward neural networks

This paper investigates the batch back-propagation algorithm with penalty for training feedforward neural networks. A usual penalty is considered, which is a term proportional to the norm of the weights. The learning rate is set to be a small constant or an adaptive series. The main contribution of this paper is to theoretically prove the boundedness of the weights in the network training process. This boundedness is then used to prove some convergence results of the algorithm, which cover both the weak and strong convergence. Simulation results are given to support the theoretical findings.