Learning algorithms utilizing quasi-geodesic flows on the Stiefel manifold

In this paper we extend the natural gradient method for neural networks to the case where the weight vectors are constrained to the Stiefel manifold. The proposed methods involve numerical integration techniques of the gradient flow without violating the manifold constraints. The extensions are based on geodesics. We rigorously formulate the previously proposed natural gradient and geodesics on the manifold exploiting the fact that the Stiefel manifold is a homogeneous space having a transitive action by the orthogonal group. Based on this fact, we further develop a simpler updating rule and one parameter family of its generalizations. The effectiveness of the proposed methods is validated by experiments in minor subspace analysis and independent component analysis.