Rejection based multipath reconstruction for background estimation in video sequences with stationary objects

•We propose a new temporal-spatial block-based Background estimation approach to compute a foreground-free image for video sequences.•Threshold-free clustering is proposed to discover similar blocks over time which contain the background data.•An iterative spatial reconstruction selects blocks to obtain the final background.•The performance improvement is validated in two datasets (36 sequences) using 13 state-of-the-art algorithms.

Background estimation in video consists in extracting a foreground-free image from a set of training frames. Moving and stationary objects may affect the background visibility, thus invalidating the assumption of many related literature where background is the temporal dominant data. In this paper, we present a temporal-spatial block-level approach for background estimation in video to cope with moving and stationary objects. First, a Temporal Analysis module obtains a compact representation of the training data by motion filtering and dimensionality reduction. Then, a threshold-free hierarchical clustering determines a set of candidates to represent the background for each spatial location (block). Second, a Spatial Analysis module iteratively reconstructs the background using these candidates. For each spatial location, multiple reconstruction hypotheses (paths) are explored to obtain its neighboring locations by enforcing inter-block similarities and intra-block homogeneity constraints in terms of color discontinuity, color dissimilarity and variability. The experimental results show that the proposed approach outperforms the related state-of-the-art over challenging video sequences in presence of moving and stationary objects.