Saliency flow based video segmentation via motion guided contour refinement

•A flexible segmentation framework is formulated as a compositional model by combining saliency map with motion prior.•A robust contour filling method is proposed based on inter-frame edge prediction regardless whether the contour areas are enclosed or not.•An efficient scheme is proposed to evaluate the reliability of motion boundaries in order to eliminate the accumulated errors when segmentation is on-the-fly.

Video object segmentation is usually regarded as a pre-process for potential performance improvement in contemporary human activity studies. Given the fact that the moving target needs to be identified and localized before any sophisticated high-level analysis being subsequently carried out, recent segmentation researches choose to achieve high accuracy by modeling a spatiotemporal scheme using moving contour detection. Unfortunately, the obtained motion boundaries are prone to bring about unreliable objects' localization due to the usage of solitude visual cues, which also motivated attemptation for more auxiliaries. Recently detection achievements have shown that saliency based high-level analysis is able to guide more reliable object localization, Thus, incorporating the saliency flow with other visual cues into an unified segmentation framework would be a promising way for more accurate video segmentation. In this paper, we formulate a unified segmentation framework based on a compositional model by combining saliency flow detection with motion estimation, and the flexibility of this framework also allows the other type of cues to be incorporated for further accuracy improvement. In addition, a robust contour filling method is proposed based on inter-frame edge prediction regardless whether the object contour is enclosed or not. With an efficient reliability evaluation and refinement of motion contours, the proposed segmentation strategy outperforms the other state-of-art approaches on two video object segmentation benchmark databases.