3D object tracking via image sets and depth-based occlusion detection

•A novel 3D Object Tracking method using Image Sets and Depth-based Occlusion Detection is proposed.•The image sets is used to address the inaccuracy of training samples during online tracking.•Depth information obtained from binocular video data is used to detect occlusion.

Object tracking has been typically formulated as an online learning problem, in which a target appearance is updated adaptively using the single-shot images tracked from the previous 2D frames. The traditional methods have the following limitations: (1) useful depth information is ignored; (2) each training and testing example is a single image, so it can be easily corrupted. In this paper, we propose a novel 3D object tracking method using image sets and depth-based occlusion detection, in which each training and testing example contains a set of image instances of an object and covers large variations in the object׳s appearance. To do so, we first represent each image set as its natural second-order statistic. Then, we use kernel partial least squares to adaptively learn low-dimensional discriminative feature subspace for object representation. Third, to prevent improper appearance model updating during occlusions, we exploit depth information obtained from binocular video data to detect occlusion. Finally, to alleviate the tracker drifting problem caused by model update, we exploit both the ground truth appearance information of the object labeled in the initial frames and the image observations obtained online. Extensive experiments on challenging video sequences demonstrate the robustness and effectiveness of the proposed method.