On the simultaneous recognition of identity and expression from BU-3DFE datasets

We propose a new linear model, based on resampling 3D face meshes to convert them to 3D matrices, to recognize identity and expression simultaneously. This contrasts with bilinear models currently used with 3D meshes. The matrices are amenable to algebraic operations for facial data analysis and synthesis. Facial emotion is represented as a linear combination of its identity and expression using principal components extracted from training data as neutral-to-emotion deformations. The linear model is applicable to other mesh data with pose variations after correction using recently available techniques. The proposed approach avoids the problem of correspondence between pairs of person’s neutral and emotion meshes for estimating facial deformations used as features. Identity and expression recognition accuracies, obtained by representing resampled matrices as linear combinations of composite depth-color (gray) PCs, are better than the results in the literature on both simultaneous identity-expression using bilinear models and expression-only recognition using deformable models, facial action codes, distances between pairs of annotated facial points as features and others. The proposed framework can also be used to generate synthetic matrices displaying a wide array of natural and mixed emotions for any chosen identity. A byproduct is the result that second-order deformations as features do not seem to perform as effectively as first-order deformations for identity and expression recognition.

► Facial emotion modeled as a linear combination of 3D matrix identity-expression training data PCs.
► Simultaneous identity and expression recognition of resampled BU-3DFE database.
► Approach distinct from bilinear models for identity and expression recognition from 3D meshes.
► Identity-expression accuracies 98.96–93.75% are better than bilinear model and expression-only results.
► Resampled matrices are amenable to algebraic operations for analysis and synthesis of faces.