Multi-view facial landmark detection by using a 3D shape model

•Accurate facial landmark detector coupled with a 6 DoF head pose estimator•A reliable global method followed by a precise local optimization•Novel learning method based on structured output SVM•Multi-view detector working on a wide range of viewing angles (frontal-profile). Self-occlusions handled.•Comparison with recent state-of-the-art methods on standard “in the wild” datasets with quantitatively favourable results.

An algorithm for accurate localization of facial landmarks coupled with a head pose estimation from a single monocular image is proposed. The algorithm is formulated as an optimization problem where the sum of individual landmark scoring functions is maximized with respect to the camera pose by fitting a parametric 3D shape model. The landmark scoring functions are trained by a structured output SVM classifier that takes a distance to the true landmark position into account when learning. The optimization criterion is non-convex and we propose a robust initialization scheme which employs a global method to detect a raw but reliable initial landmark position. Self-occlusions causing landmarks invisibility are handled explicitly by excluding the corresponding contributions from the data term. This allows the algorithm to operate correctly for a large range of viewing angles. Experiments on standard “in-the-wild” datasets demonstrate that the proposed algorithm outperforms several state-of-the-art landmark detectors especially for non-frontal face images. The algorithm achieves the average relative landmark localization error below 10% of the interocular distance in 98.3% of the 300 W dataset test images.