Coarse-to-fine outlier correction with applications in structure from motion

•We propose an outlier detection and correction framework in structure from motion with the aim of handling outliers and missing data.•This method lies on estimating the fidelity of the estimated camera motion.•The Huber M-estimator is extended into the matrix form.•The estimated fidelity is incorporated into the extension of Huber M-estimator to detect and correct outliers.

This paper presents a robust method for outlier detection and correction in structure from motion. Aiming at handling outliers together with missing data, the Discrete Cosine Transform (DCT) based Column Space Fitting (CSF) algorithm is extended and improved. The use of the DCT basis allows for a coarse-to-fine optimization strategy that reconstructs 3D scene geometry and camera motion by increasing the number of DCT basis vectors. With a certain DCT basis, an interior point based L1-norm solver is used to successively estimate 3D scene structure. In addition, the fidelity of the estimated camera motion matrix is first integrated into an extension of Huber M-estimator to find outliers and to robustly estimate the update magnitude for each outlier. This fidelity can be measured by the effects of camera motion matrix on re-projection errors. Because the Huber M-estimator is only applicable to vector, we extend it into the matrix form. With the increase in the number of DCT basis vectors, outliers are corrected in a coarse-to-fine manner. Experiments on both synthetic and real image sequences confirm the effectiveness of the proposed method.