Joint Depth and Semantic Inference from a Single Image via Elastic Conditional Random Field

The estimations of depth and regional semantics from a single image have traditionally been considered as two separated problems. In this paper, we argue that these two tasks provide complementary information, which therefore can be performed jointly to reinforce individual tasks in terms of both accuracy and speed. In particular, we propose an Elastic Conditional Random Field (E-CRF) deployed upon superpixel segmentations, which models the interdependency between depth and semantics to refine each other in an iterative manner. Differing from the traditional CRFs, E-CRF makes edges elastically hidden/emergent during inference to conduct fast Loopy Belief Propagation, while explicitly modeling the depth-label interdependency to achieve high inference accuracy. Moreover, the Structured Support Vector Machine (SSVM) is further introduced to drastically speed up the inference. We have conducted extensive evaluations on both Make3D and NYU benchmark datasets, which demonstrated that our E-CRF method significantly outperforms state-of-the-art techniques in terms of precision, while significantly accelerating the inference speed (2-3 orders of magnitude).