Reconstruction of non-rigid 3D shapes from stereo-motion

Several non-rigid structure from motion methods have been proposed so far in order to recover both the motion and the non-rigid structure of an object. However, these monocular algorithms fail to give reliable 3D shape estimates when the overall rigid motion of the sequence is small. Aiming to overcome this limitation, in this paper we propose a novel approach for the 3D Euclidean reconstruction of deformable objects observed by an uncalibrated stereo rig. Using a stereo setup drastically improves the 3D model estimation when the observed 3D shape is mostly deforming without undergoing strong rigid motion. Our approach is based on the following steps. Firstly, the stereo system is automatically calibrated and used to compute metric rigid structures from pairs of views. Afterwards, these 3D shapes are aligned to a reference view using a RANSAC method in order to compute the mean shape of the object and to select the subset of points which have remained rigid throughout the sequence. The selected rigid points are then used to compute frame-wise shape registration and to robustly extract the motion parameters from frame to frame. Finally, all this information is used as initial estimates of a non-linear optimization which allows us to refine the initial solution and also to recover the non-rigid 3D model. Exhaustive results on synthetic and real data prove the performance of our proposal estimating motion, non-rigid models and stereo camera parameters even when there is no rigid motion in the original sequence.

Research highlights
► A novel approach for the 3D Euclidean reconstruction of non-rigid objects observed by a stereo rig.
► Results on synthetic and real data prove that the use of a stereo rig allows to recover reliable 3D shape estimates.
► A comparison with 2 monocular structure from motion methods, demonstrate the validity of our approach when the object is not doing any rigid motion during the sequence.