Identifying multiple objects from their appearance in inaccurate detections

• Localize and learn appearances of targets from object detections in video.
• Associate targets to features instead of detections to deal with partial occlusions.
• Post-processing provides pixel-level segmentation of individual objects.
• Scenarios can have inaccurate detections, occlusions, erratic motion, frame skips.
• Outperforms state-of-the-art batch and online multi-view trackers on fight scenes.

We propose a novel method for keeping track of multiple objects in provided regions of interest, i.e. object detections, specifically in cases where a single object results in multiple co-occurring detections (e.g. when objects exhibit unusual size or pose) or a single detection spans multiple objects (e.g. during occlusion). Our method identifies a minimal set of objects to explain the observed features, which are extracted from the regions of interest in a set of frames. Focusing on appearance rather than temporal cues, we treat video as an unordered collection of frames, and “unmix” object appearances from inaccurate detections within a Latent Dirichlet Allocation (LDA) framework, for which we propose an efficient Variational Bayes inference method. After the objects have been localized and their appearances have been learned, we can use the posterior distributions to “back-project” the assigned object features to the image and obtain segmentation at pixel level. In experiments on challenging datasets, we show that our batch method outperforms state-of-the-art batch and on-line multi-view trackers in terms of number of identity switches and proportion of correctly identified objects. We make our software and new dataset publicly available for non-commercial, benchmarking purposes.