High-resolution 3D surface strain magnitude using 2D camera and low-resolution depth sensor

•Measures the 3-D surface strain impacted on the face during facial expressions.•Describes an automatic approach for calibrating Kinect with external camera.•Method shows high correlation between 3-D strain maps calculated from two views.•Method robust to multiple depth resolutions.•Over 100 subjects and 600 expressions used for testing.

Generating 2-D strain maps of the face during facial expressions provides a useful feature that captures the bio-mechanics of facial skin tissue, and has had wide application in several research areas. However, most applications have been restricted to collecting data on a single pose. Moreover, methods that strictly use 2-D images for motion estimation can potentially suppress large strains because of projective distortions caused by the curvature of the face. This paper proposes a method that allows estimation of 3-D surface strain using a low-resolution depth sensor. The algorithm consists of automatically aligning a rough approximation of a 3-D surface with an external high resolution camera image. We provide experimental results that demonstrate the robustness of the method on a dataset collected using the Microsoft Kinect synchronized with two external high resolution cameras, as well as 101 subjects from a publicly available 3-D facial expression video database (BU4DFE).