3D-2D face recognition with pose and illumination normalization

- Describes a conceptual framework for 3D-2D (or 2D-3D) face recognition.
- Proposes a novel 3D-2D system for 2D image face recognition from 3D datasets.
- Proposes a method to build subject-specific 3D gallery models, using 3D+2D data, and a method for model-based, texture representation and relighting.
- 3D-2D recognition surpasses 2D-2D on challenging 2D+3D data with pose and illumination variations, and can approximate 3D-3D, shape-based similarity methods.
- Representation and normalization using 3D models can compensate for non-frontal poses or different lighting conditions.

In this paper, we propose a 3D-2D framework for face recognition that is more practical than 3D-3D, yet more accurate than 2D-2D. For 3D-2D face recognition, the gallery data comprises of 3D shape and 2D texture data and the probes are arbitrary 2D images. A 3D-2D system (UR2D) is presented that is based on a 3D deformable face model that allows registration of 3D and 2D data, face alignment, and normalization of pose and illumination. During enrollment, subject-specific 3D models are constructed using 3D+2D data. For recognition, 2D images are represented in a normalized image space using the gallery 3D models and landmark-based 3D-2D projection estimation. A method for bidirectional relighting is applied for non-linear, local illumination normalization between probe and gallery textures, and a global orientation-based correlation metric is used for pairwise similarity scoring. The generated, personalized, pose- and light- normalized signatures can be used for one-to-one verification or one-to-many identification. Results for 3D-2D face recognition on the UHDB11 3D-2D database with 2D images under large illumination and pose variations support our hypothesis that, in challenging datasets, 3D-2D outperforms 2D-2D and decreases the performance gap against 3D-3D face recognition. Evaluations on FRGC v2.0 3D-2D data with frontal facial images, demonstrate that the method can generalize to databases with different and diverse illumination conditions.