Synthetic aperture photography using a moving camera-IMU system

•The first method that uses camera and IMU data to generate synthetic aperture images.•Analytical factorization of the problem into two parts and solved by separate data.•Totally scene independent homography estimation based on IMU data.•Avoids complicated optimization and reduces computational complexity.•Provides user a more flexible and inexpensive system than conventional methods.

Occlusion poses as a critical challenge in computer vision for a long time. Recently, the technique of synthetic aperture photography using a camera array has been regarded as a promising way to address the problem of occluded object imaging. Unfortunately, the expensive cost of a standard camera array system with the required calibration procedure still limits the widespread popularity of this technique. To make synthetic aperture photography practical, in this paper, we propose a novel approach based on a mobile camera-IMU system moving on a planar surface. The contributions of our work include: (1) with geometric property assumptions, we mathematically prove that the transformation matrices for synthetic aperture image generation can be factorized into an infinite homography and a shift; (2) the IMU data are utilized to estimate the homography by selecting the plane at infinity as the reference plane, which is scene independent and does not require any calibration; (3) compared to the conventional camera array based approaches, the proposed method effectively improves the solution for synthetic aperture images capturing and generation by using a more flexible, convenient and inexpensive system. Extensive experimental results with qualitative and quantitative evaluations have demonstrated the see-through-occlusion performance with satisfactory accuracy of the proposed approach in different scenarios.