Spectral embedding based facial expression recognition with multiple features

• Spectral embedding based feature fusion framework is proposed to combine the appearance based and geometry based features for facial expression recognition.
• A supervised multi-view spectral embedding algorithm is developed to achieve more discriminative embedding.
• In order to solve the out-of-sample problem, we utilize a linearization method to map unseen data to the unified low dimensional subspace discovered by the MSE algorithm.
• We perform a comprehensive study of five widely used facial expression features, including AAM, LBP, Multiscale WLD, SIFT descriptor and Gabor filters.

Many approaches to facial expression recognition utilize only one type of features at a time. It can be difficult for a single type of features to characterize in a best possible way the variations and complexity of realistic facial expressions. In this paper, we propose a spectral embedding based multi-view dimension reduction method to fuse multiple features for facial expression recognition. Facial expression features extracted from one type of expressions can be assumed to form a manifold embedded in a high dimensional feature space. We construct a neighborhood graph that encodes the structure of the manifold locally. A graph Laplacian matrix is constructed whose spectral decompositions reveal the low dimensional structure of the manifold. In order to obtain discriminative features for classification, we propose to build a neighborhood graph in a supervised manner by utilizing the label information of training data. As a result, multiple features are able to be transformed into a unified low dimensional feature space by combining the Laplacian matrix of each view with the multiview spectral embedding algorithm. A linearization method is utilized to map unseen data to the learned unified subspace. Experiments are conducted on a set of established real-world and benchmark datasets. The experimental results provide a strong support to the effectiveness of the proposed feature fusion framework on realistic facial expressions.