Keyframe-based recognition and localization during video-rate parallel tracking and mapping

Generating situational awareness by augmenting live imagery with collocated scene information has applications from game-playing to military command and control. We propose a method of object recognition, reconstruction, and localization using triangulation of SIFT features from keyframe camera poses in a 3D map. The map and keyframe poses themselves are recovered at video-rate by bundle adjustment of FAST image features in the parallel tracking and mapping algorithm. Detected objects are automatically labeled on the user's display using predefined annotations. Experimental results are given for laboratory scenes, and in more realistic applications.

Figure optionsDownload high-quality image (173 K)Download as PowerPoint slideResearch highlights
► Camera pose tracked at frame rate using FAST features.
► Live bundle adjustment optimizes 3D map and keyframe camera poses.
► SIFT features computed in keyframes and objects recognized.
► Features on recognized objects matched between keyframes, and structure of objects recovered by triangulation.
► Method demonstrated in augmented reality scenarios.