An effective methodology for dynamic 3D facial expression retrieval

• We illustrate a novel retrieval methodology for dynamic sequences of 3D face scans.
• We present a detailed evaluation of the introduced retrieval scheme.
• BU-4DFE and BP4D-Spontaneous data sets were used for experiments.
• Retrieval results are used to achieve unsupervised facial expression recognition.
• The presented results outperform state-of-the-art retrieval schemes.

The problem of facial expression recognition in dynamic sequences of 3D face scans has received a significant amount of attention in the recent past whereas the problem of retrieval in this type of data has not. A novel retrieval methodology for such data is introduced in this paper. The proposed methodology automatically detects specific facial landmarks and uses them to create a descriptor. This descriptor is the concatenation of three sub-descriptors which capture topological as well as geometric information of the 3D face scans. The motivation behind the proposed hybrid facial expression descriptor is the fact that some facial expressions, like happiness and surprise, are characterized by obvious changes in the mouth topology while others, like anger, fear and sadness, produce geometric but no significant topological changes. The proposed retrieval scheme exploits the Dynamic Time Warping technique in order to compare descriptors corresponding to different 3D facial sequences. A detailed evaluation of the introduced retrieval scheme is presented showing that it outperforms previous state-of-the-art retrieval schemes. Experiments have been conducted using the six prototypical expressions of the standard dataset BU-4DFE and the eight prototypical expressions of the recently available dataset BP4D-Spontaneous. Finally, a majority voting scheme based on the retrieval results is used to achieve unsupervised dynamic 3D facial expression recognition. The achieved classification accuracy is comparable to the state-of-the-art supervised dynamic 3D facial expression recognition techniques.