Joint facial expression recognition and intensity estimation based on weighted votes of image sequences

Facial behavior consists of dynamically changing properties of facial features as a result of muscle activation. Facial behavior analysis is a challenging problem due to complexity of emotions and variability of the facial expressions associated with the emotions. Most facial expression recognition systems attempt to recognize facial expressions without taking into account the intensity of the expressions. In this paper, a novel framework for facial expression recognition and intensity estimation with low computational complexity requirement is proposed. The algorithm constructs a representation of facial features based on a weighted voting scheme and employs Hidden Markov Models to classify an input video into one of the six basic expressions, namely anger, disgust, fear, happiness, sadness, and surprise. The temporal segments, neutral, onset, and apex, of an expression are then obtained by means of a change-point detector. Evaluations on subject-independent analysis was conducted using Cohn-Kanade dataset and Beihang University facial expression datasets. The proposed approach has demonstrated a superior performance in recognizing facial expressions and estimating expression intensities.