A spatial-temporal framework based on histogram of gradients and optical flow for facial expression recognition in video sequences

• Proposes a facial expression recognition framework that uses dynamic information.
• Dynamic information is variation in facial shape and movement of facial landmarks.
• Introduces PHOG_TOP to capture variation of facial shape in temporal domain.
• Introduces dense optical flow on a grid to detect movement of landmarks.
• Fusion of PHOG_TOP and dense optical flow for high recognition rate.

Facial expression causes different parts of the facial region to change over time and thus dynamic descriptors are inherently more suitable than static descriptors for recognising facial expressions. In this paper, we extend the spatial pyramid histogram of gradients to spatio-temporal domain to give 3-dimensional facial features and integrate them with dense optical flow to give a spatio-temporal descriptor which extracts both the spatial and dynamic motion information of facial expressions. A multi-class support vector machine based classifier with one-to-one strategy is used to recognise facial expressions. Experiments on the CK+ and MMI datasets using leave-one-out cross validation scheme demonstrate that the integrated framework achieves a better performance than using individual descriptor separately. Compared with six state of the art methods, the proposed framework demonstrates a superior performance.