Augmenting analytic SFM filters with frame-to-frame features

• An approach to incorporate frame-to-frame features within online SFM estimators.
• Like inertial measurements, such features further constrain the camera velocity.
• The proposed approach can be easily retrofitted within existing SFM estimators.
• The computational complexity is reduced from cubic initially to linear.

In Structure From Motion (SFM), image features are matched in either an extended number of frames or only in pairs of consecutive frames. Traditionally, SFM filters have been applied using only one of the two matching paradigms, with the Long Range (LR) feature technique being more popular because of the fact that features that are matched across multiple frames provide stronger constraints on structure and motion. Nevertheless, Frame-to-Frame (F2F) features possess the desirable property of being abundant because of the large similarity that exists between closely spaced frames. Although the use of such features has been limited mostly to the determination of inter-frame camera motion, we argue that significant improvements can be attained in online filter-based SFM by integrating the F2F features into filters that use LR features. The main contributions of this paper are twofold. First, it presents a new method that enables the incorporation of F2F information in any analytical filter in a fashion that requires minimal change to the existing filter. Our results show that by doing so, large increases in accuracy are achieved in both the structure and motion estimates. Second, thanks to mathematical simplifications we realize in the filter, we minimize the computational burden of F2F integration by two orders of magnitude, thereby enabling its real-time implementation. Experimental results on real and simulated data prove the success of the proposed approach.