Correspondence as energy-based segmentation

We pose the correspondence problem as one of energy-based segmentation. In this framework, correspondence assigns each pixel in an image to exactly one of several non-overlapping regions, and it also computes a displacement function for each region. The framework is better able to capture the scene geometry than the more direct formulation of matching pixels in two or more images, particularly when the surfaces in the scene are not fronto-parallel. To illustrate the framework, we present a specific correspondence algorithm that minimizes an energy functional by alternating between (1) segmenting the image into a number of non-overlapping regions using the multiway-cut algorithm of Boykov, Veksler, and Zabih; and (2) finding the affine parameters describing the displacement of the pixels in each region. After convergence, a final step escapes local minima due to over-segmentation. The basic algorithm is extended in two ways: using ground control points to detect long, thin regions; and warping segmentation results to efficiently process image sequences. Experiments on real images show the algorithm’s ability to find an accurate segmentation and displacement map, as well as discontinuities and creases, on a wide variety of stereo and motion imagery.