Unsupervised segmentation of highly dynamic scenes through global optimization of multiscale cues

•We introduce a global optimization of weighted total variation energy functional.•The energy combines motion and spectral boundaries with object inside mappings.•Alternating direction convex optimization provides high-quality salient mapping.•Integrating mapping with MRF facilitates sequential combination of multiscale cues.•Feasibility and superiority are demonstrated in segmenting highly dynamic scenes.

We propose a novel method for highly dynamic scene segmentation by formulating foreground object extraction as a global optimization framework that integrates a set of multiscale spatio-temporal cues. The multiscale features consist of a combination of motion and spectral components at a pixel level as well as spatio-temporal consistency constraints between superpixels. To compensate for the ambiguities of foreground hypothesis due to highly dynamic and cluttered backgrounds, we formulate salient foreground mapping as a convex optimization of weighted total variation energy, which is efficiently solved by using an alternating minimization scheme. Moreover, the appearance and position spatio-temporal consistency constraints between superpixels are explicitly incorporated into a Markov random field energy functional for further refinement of the set of salient pixel-level foreground mapping. This work facilitates sequential integration of multiscale probability constraints into a global optimal segmentation framework to help address object boundary ambiguities in the case of highly dynamic scenes. Extensive experiments on challenging dynamic scene data sets demonstrate the feasibility and superiority of the proposed segmentation approach.