Smooth adaptive fitting of 3D face model for the estimation of rigid and nonrigid facial motion in video sequences

We propose a 3D wireframe face model alignment for the task of simultaneously tracking of rigid head motion and nonrigid facial expressions in video sequences. The system integrates two levels: (i) at the low level, automatic and accurate location of facial features are obtained via a cascaded optimization algorithm of a 2D shape model, (ii) at the high level, we recover, via minimizing an energy function, the optimal motion parameters of the 3D model, namely the 3D rigid motion parameters and seven nonrigid animation (Action Unit) parameters. In this latter inference, a 3D face shape model (Candide) is automatically fitted to the image sequence via a least squares minimization of the energy, defined as the residual between the projected 3D wireframe model and the 2D shape model, meanwhile imposing temporal and spatial motion-smoothness constraints over the 3D model points. Our proposed system tackles many disadvantages of the optimization and training associated with active appearance models. Extensive fitting and tracking experiments demonstrate the feasibility, accuracy and effectiveness of the developed methods. Qualitative and quantitative performance of the proposed system on several facial sequences, indicate its potential usefulness for multimedia applications, as well as facial expression analysis.

► Accurate facial feature tracking leads to robust animation parameter estimation. ► Energy optimization of 3D facial expression improves the smoothness of the estimate. ► 3D after 2D face motion estimation from video forms a basis for emotion analysis.