Lighting-aware face frontalization for unconstrained face recognition

Face recognition under variable pose and lighting is still one of the most challenging problems, despite the great progress achieved in unconstrained face recognition in recent years. Pose variation is essentially a misalignment problem together with invisible region caused by self-occlusion. In this paper, we propose a lighting-aware face frontalization method that aims to generate both lighting-recovered and lighting-normalized frontalized images, based on only five fiducial landmarks. Basic frontalization is first performed by aligning a generic 3D face model into the input face and rendering it at frontal pose, with an accurate visible region estimation based on face borderline detection. Then we apply the illumination-invariant quotient image, estimated from the visible region, as a face symmetrical feature to fill the invisible region. Lighting-recovered face frontalization (LRFF) is conducted by rendering the estimated lighting on the invisible region. By adjusting the combination parameters, lighting-normalized face frontalization (LNFF) is performed by rendering the canonical lighting on the face. Although its simplicity, our LRFF method competes well with more sophisticated frontalization techniques, on the experiments of LFW database. Moreover, combined with our recently proposed LRA-based classifier, the LNFF based method outperforms the deep learning based methods by about 6% on the challenging experiment on Multiple PIE database under variable pose and lighting.