Learning structured and non-redundant representations with deep neural networks

This paper proposes a novel regularizer named Structured Decorrelation Constraint, to address both the generalization and optimization of deep neural networks, including multiple-layer perceptrons and convolutional neural networks. Our proposed regularizer reduces overfitting by breaking the co-adaptions between the neurons with an explicit penalty. As a result, the network is capable of learning non-redundant representations. Meanwhile, the proposed regularizer encourages the networks to learn structured high-level features to aid the networks' optimization during training. To this end, neurons are constrained to behave obeying a group prior. Our regularizer applies to various types of layers, including fully connected layers, convolutional layers and normalization layers. The loss of our regularizer can be directly minimized along with the network's classification loss by stochastic gradient descent. Experiments show that the proposed regularizer obviously relieves the overfitting problem of the existing deep networks. It yields much better performance on extensive datasets than the conventional regularizers like Dropout.