Hallucination of facial details from degraded images using 3D face models

•A 3D model-based algorithm for face hallucination at any pose and illumination•A method for including non-local effects (e.g. blur) in 3D analysis-by-synthesis•The algorithm combines low spatial frequency information with details of a 3D model.•Transfer of high spatial frequency details for hallucination on the level of pores•Occlusion handling and seamless texture reconstruction

The goals of this paper are: (1) to enhance the quality of images of faces, (2) to enable 3D Morphable Models (3DMMs) to cope with severely degraded images, and (3) to reconstruct textured 3D faces with details that are not in the input images. Details that are lost in the input images due to blur, low resolution or occlusions, are filled in by the 3DMM and an additional texture enhancement algorithm that adds high-resolution details from example faces. By leveraging class-specific knowledge, this restoration process goes beyond what general image operations such as deblurring or inpainting can achieve. The benefit of the 3DMM for image restoration is that it can be applied to any pose and illumination, unlike image-based methods. However, it is only with the new fitting algorithm that 3DMMs can produce realistic faces from severely degraded images. The new method includes the blurring or downsampling operator explicitly into the analysis-by-synthesis algorithm.

Graphical abstractFigure optionsDownload full-size imageDownload high-quality image (97 K)Download as PowerPoint slide