Real-time interactive modeling and scalable multiple object tracking for AR

We propose a real-time solution for modeling and tracking multiple 3D objects in unknown environments for Augmented Reality. The proposed solution consists of both scalable tracking and interactive modeling. Our contribution is twofold: First, we show how to scale with the number of objects using keyframes. This is done by combining recent techniques for image retrieval and online Structure from Motion, which can be run in parallel. As a result, tracking 50 objects in 3D can be done within 6–35 ms per frame, even under difficult conditions for tracking. Second, we propose a method to let the user add new objects very quickly. The user simply has to select in an image a 2D region lying on the object. A 3D primitive is then fitted to the features within this region, and adjusted to create the object 3D model. We demonstrate the modeling of polygonal and circular-based objects. In practice, this procedure takes less than a minute.

Graphical abstractFigure optionsDownload full-size imageDownload high-quality image (364 K)Download as PowerPoint slideHighlights► We propose a real-time modeling and tracking of multiple objects for augmented reality. ► Our real-time Structure from Motion-based modeling approach allowed various types of objects modeling, such as boxes, cylinders, and so on. ► We simultaneously detected and tracked 14 objects in real-time when 50 objects are in a database.