Automatic facial expression recognition based on spatiotemporal descriptors

Facial expression’s machine analysis is one of the most challenging problems in Human–Computer Interaction (HCI). Naturally, facial expressions depend on subtle movements of facial muscles to show emotional states. After having studied the relations between basic expressions and corresponding facial deformation models, we propose two new textons, VTB and moments on spatiotemporal plane, to describe the transformation of human face during facial expressions. These descriptors aim at catching both general shape changes and motion texture details. Therefore, modeling the temporal behavior of facial expression captures the dynamic deformation of facial components. Finally, SVM based system is used to efficiently recognize the expression for a single image in sequence. Then, the probabilities of all the frames are used to predict the class of the current sequence. The experimental results are evaluated on both Cohan–Kanade and MMI databases. By comparison to other methods, the effectiveness of our method is clearly demonstrated.

► New texton VTB (Vertical-Time-Backward) on spatiotemporal planes to capture the vertical movements.
► Unique usage of moments on spatiotemporal domain to capture the dynamic deformation of facial components.
► Using probabilities from image based method to sequence based method.