Hierarchical On-line Appearance-Based Tracking for 3D head pose, eyebrows, lips, eyelids and irises

In this paper, we propose an On-line Appearance-Based Tracker (OABT) for simultaneous tracking of 3D head pose, lips, eyebrows, eyelids and irises in monocular video sequences. In contrast to previously proposed tracking approaches, which deal with face and gaze tracking separately, our OABT can also be used for eyelid and iris tracking, as well as 3D head pose, lips and eyebrows facial actions tracking. Furthermore, our approach applies an on-line learning of changes in the appearance of the tracked target. Hence, the prior training of appearance models, which usually requires a large amount of labeled facial images, is avoided. Moreover, the proposed method is built upon a hierarchical combination of three OABTs, which are optimized using a Levenberg–Marquardt Algorithm (LMA) enhanced with line-search procedures. This, in turn, makes the proposed method robust to changes in lighting conditions, occlusions and translucent textures, as evidenced by our experiments. Finally, the proposed method achieves head and facial actions tracking in real-time.

Graphical abstractFigure optionsDownload full-size imageDownload high-quality image (199 K)Download as PowerPoint slideHighlights► 3D head pose, eyebrows, lips, eyelids and irises are tracked simultaneously. ► Monocular video sequences are tracked in real-time, no color information required. ► Line-search, backtracking and Levenberg–Marquardt algorithms ensure convergence. ► Illumination changes, occlusions and drifting challenges are robustly handled. ► Variety of sequences, facial expressions, occlusions and lighting are tested.