Integrating tracking with fine object segmentation

•A novel method for joint tracking and fine object segmentation in videos•Efficient integration of EM-based tracking and Random Walker-based image segmentation•No strong constraints are imposed on the target appearance or the camera motion.•Experimental evaluation against a large collection of state-of-the-art methods•Explicit and efficient fine object segmentation facilitates drift-free tracking.

We present a novel method for on-line, joint object tracking and segmentation in a monocular video captured by a possibly moving camera. Our goal is to integrate tracking and fine segmentation of a single, previously unseen, potentially non-rigid object of unconstrained appearance, given its segmentation in the first frame of an image sequence as the only prior information. To this end, we tightly couple an existing kernel-based object tracking method with Random Walker-based image segmentation. Bayesian inference mediates between tracking and segmentation, enabling effective data fusion of pixel-wise spatial and color visual cues. The fine segmentation of an object at a certain frame provides tracking with reliable initialization for the next frame, closing the loop between the two building blocks of the proposed framework. The effectiveness of the proposed methodology is evaluated experimentally by comparing it to a large collection of state of the art tracking and video-based object segmentation methods on the basis of a data set consisting of several challenging image sequences for which ground truth data is available.