Online domain-shift learning and object tracking based on nonlinear dynamic models and particle filters on Riemannian manifolds

• Online domain-shift learning of dynamic object appearances with out-of-plane motions.
• Bayesian formulation on Riemannian manifolds using nonlinear state space models.
• Two particle filters, one on manifolds for appearance learning, another for object tracking.
• Occlusion handling to prevent online learning when objects are likely occluded.
• Tracking scheme is applied to visual-band videos and thermal infrared videos.

This paper proposes a novel online domain-shift appearance learning and object tracking scheme on a Riemannian manifold for visual and infrared videos, especially for video scenarios containing large deformable objects with fast out-of-plane pose changes that could be accompanied by partial occlusions. Although Riemannian manifolds and covariance descriptors are promising for visual object tracking, the use of Riemannian mean from a window of observations, spatially insensitive covariance descriptors, fast significant out-of-plane (non-planar) pose changes, and long-term partial occlusions of large-size deformable objects in video limits the performance of such trackers. The proposed method tackles these issues with the following main contributions: (a) Proposing a Bayesian formulation on Riemannian manifolds by using particle filters on the manifold and using appearance particles in each time instant for computing the Riemannian mean, rather than using a window of observations. (b) Proposing a nonlinear dynamic model for online domain-shift learning on the manifold, where the model includes both manifold object appearance and its velocity. (c) Introducing a criterion-based partial occlusion handling approach in online learning. (d) Tracking object bounding box by using affine parametric shape modeling with manifold appearance embedded. (e) Incorporating spatial, frequency and orientation information in the covariance descriptor by extracting Gabor features in a partitioned bounding box. (f) Effectively applying to both visual-band videos and thermal-infrared videos. To realize the proposed tracker, two particle filters are employed: one is applied on the Riemannian manifold for generating candidate appearance particles and another is on vector space for generating candidate box particles. Further, tracking and online learning are performed in alternation to mitigate the tracking drift. Experiments on both visual and infrared videos have shown robust tracking performance of the proposed scheme. Comparisons and evaluations with ten existing state-of-art trackers provide further support to the proposed scheme.