3D Active Appearance Model alignment using intensity and range data

•Align a 3D AAM simultaneously to a range and intensity image.•Weight 2D and 3D data based on images and the sensor characteristics.•We achieve stable alignment under head rotations of up to 80°80°.•Experimentally validated with two different sets of depth and intensity cameras.

Active Appearance Models (AAMs) are widely used to match a shape and appearance model to an image. This paper extends the commonly used 2D shape model to 3D, and introduces an effective method for integrating alignment to RGB and 3D range images. The use of a three dimensional model allows accurate estimation of head orientation, shape and position. Existing approaches combining range and intensity data use a manually tuned weighting function to balance 2D and 3D alignments. We develop a method to guide the alignment based on the observed image properties and the sensor characteristics. Our approach is experimentally validated using two different sets of depth and RGB cameras. In our experiments we achieve stable alignment under wide angular head rotations of up to 80°80° with a maximum improvement of 26% compared to the 3D AAM using intensity image and 30% improvement over the state-of-the-art 3DMM methods in terms of 3D head pose estimation.