A combined post-filtering method to improve accuracy of variational optical flow estimation

• We propose a multi-scale and nonlinear structure tensor based 3D spatial-scale Harris edge detector for edges of flow fields.
• We propose a novel hybrid gradient bilateral and Gaussian filter approach through a spatial-scale gradient signal-to-noise ratio segmentation.
• We present a segmentation method to classify the structure tensor elements into continuity and non-discontinuity regions.
• A piecewise occlusions detection approach is used to detect occlusions of the flow field.
• We propose a combined post-filtering method with the weighted median filter, bilateral filter, and the fast median filter.

We present a novel combined post-filtering (CPF) method to improve the accuracy of optical flow estimation. Its attractive advantages are that outliers reduction is attained while discontinuities are well preserved, and occlusions are partially handled. Major contributions are the following: First, the structure tensor (ST) based edge detection is introduced to extract flow edges. Moreover, we improve the detection performance by extending the traditional 2D spatial edge detector into spatial-scale 3D space, and also using a gradient bilateral filter (GBF) to replace the linear Gaussian filter to construct a multi-scale nonlinear ST. GBF is useful to preserve discontinuity but it is computationally expensive. A hybrid GBF and Gaussian filter (HGBGF) approach is proposed by means of a spatial-scale gradient signal-to-noise ratio (SNR) measure to solve the low efficiency issue. Additionally, a piecewise occlusion detection method is used to extract occlusions. Second, we apply a CPF method, which uses a weighted median filter (WMF), a bilateral filter (BF) and a fast median filter (MF), to post-smooth the detected edges and occlusions, and the other flat regions of the flow field, respectively. Benchmark tests on both synthetic and real sequences demonstrate the effectiveness of our method.