Combined segmentation, reconstruction, and tracking of multiple targets in multi-view video sequences

•Addressing the problem of Joint segmentation, reconstruction and tracking of multiple targets from multi-view videos.•Casting the problem as data association among extracted superpixels from images.•Optimizing a flow graph to solve the global data association in order to segment and reconstruct targets.•Fast obtaining the solution of graph by performing two stages of optimization.•Conduction experimental results on known public datasets and analyzing the proposed algorithm.

Tracking of multiple targets in a crowded environment using tracking by detection algorithms has been investigated thoroughly. Although these techniques are quite successful, they suffer from the loss of much detailed information about targets in detection boxes, which is highly desirable in many applications like activity recognition. To address this problem, we propose an approach that tracks superpixels instead of detection boxes in multi-view video sequences. Specifically, we first extract superpixels from detection boxes and then associate them within each detection box, over several views and time steps that lead to a combined segmentation, reconstruction, and tracking of superpixels. We construct a flow graph and incorporate both visual and geometric cues in a global optimization framework to minimize its cost. Hence, we simultaneously achieve segmentation, reconstruction and tracking of targets in video. Experimental results confirm that the proposed approach outperforms state-of-the-art techniques for tracking while achieving comparable results in segmentation.