2D facial expression recognition via 3D reconstruction and feature fusion

• This paper proposed a method for facial expression recognition.
• In proposed method, facial depth has been added to facial texture for feature extraction.
• We demonstrated that adding the facial depth to feature extraction is effective.
• The 3DH-LLBP is proposed for feature extraction from facial depth images.
• 2D facial expression recognition is performed by combination of 2D and 3D features through feature fusion.

In this paper, a novel feature extraction method is proposed for facial expression recognition by extracting the feature from facial depth and 3D mesh alongside texture. Accordingly, the 3D Facial Expression Generic Elastic Model (3D FE-GEM) method is used to reconstruct an expression-invariant 3D model from the human face. Then, the texture, depth and mesh are extracted from the reconstructed face model. Afterwards, the Local Binary Pattern (LBP), proposed 3D High-Low Local Binary Pattern (3DH-LLBP) and Local Normal Binary Patterns (LNBPs) are applied to texture, depth and mesh of the face, respectively, to extract the feature from 2D images. Finally, the final feature vectors are generated through feature fusion and are classified by the Support Vector Machine (SVM). Convincing results are acquired for facial expression recognition on the CK+, CK, JAFFE and Bosphorus image databases compared to several state-of-the-art methods.