Face identification with second-order pooling in single-layer networks

Automatic face recognition has received significant performance improvement by developing specialized facial image representations. On the other hand, spatial pyramid pooling of features encoded by an over-complete dictionary has been the key component of many state-of-the-art generic objective classification systems. Inspired by its success, in this work we develop a new face image representation method under the framework of single-layer networks, where the key component is the second-order pooling layer. The proposed method differs from the previous methods in that, we encode the densely extracted local patches by a small-size dictionary; and the facial image signatures are obtained by pooling the second-order statistics of the encoded features. We show the importance of the encoding procedure, which is bypassed by the original second-order pooling method to avoid the high computational cost. Equipped with a simple linear classifier, the proposed method outperforms the state-of-the-art face identification performance by large margins. For example, on the LFW databases, the proposed method performs better than the previous best by around 13% accuracy.