Robust bilayer video segmentation by adaptive propagation of global shape and local appearance

Segmenting semantic objects of interest from video has long been an active research topic, with a wide range of potential applications. In this paper, we present a bilayer video segmentation method robust to abrupt motion and change in appearance for both the foreground and background. Specifically, based on a few manually segmented keyframes, the proposed method propagates the global shape of the foreground as priors to adjacent frames by applying branch-and-mincut [1], which jointly estimates what is optimal among a set of shapes along with its pose and the corresponding segmentation in the current image. Based on this preliminary segmentation we determine two types of local regions likely to have erroneous results, and apply a probabilistic framework where shape and appearance cues are adaptively emphasized for local refinement. With each successive frame segmentation, the set of shapes applied as priors are incrementally updated. Experimental results support the robustness of the proposed method for obstacles such as background clutter, motion, and appearance changes, from only a small number of user segmented keyframes.