Recovering an indoor 3D layout with top-down semantic segmentation from a single image

•We recover a 3D layout of an indoor image with top-down features.•We combine detections into a joint semantic segmentation and layout estimation.•Our features complement the limitations of existing bottom-up geometric features.•We achieved the state-of-the-art layout accuracy on the UIUC indoor dataset.

In this paper, we propose a framework to recover a 3D cuboidal indoor scene with a novel detector-based semantic segmentation feature and a carefully-modified orientation map. We use those features to mimic the ability of humans to recognize a 3D layout from a single image. We define all the potentials in our model under a conditional random field formulation. Our experimental results show the effectiveness of our new features which complement the limitations of existing bottom-up geometric features while achieving the state-of-the-art layout accuracy on the indoor UIUC dataset.