Contextual object category recognition for RGB-D scene labeling

• We use a classifier based on a novel fusion of feature vectors (the VFH-Texton).
• We derive an object-to-object context MRF model based on Flickr label co-occurrence data.
• We investigate the model’s parameters’ convergence as a function of Flickr’s sample size.
• We train the system on the RGB-D Object Dataset and test on the NYU Dataset as well.
• Finally we illustrate an increasing performance through the use of the MRF.

Recent advances in computer vision on the one hand, and imaging technologies on the other hand, have opened up a number of interesting possibilities for robust 3D scene labeling. This paper presents contributions in several directions to improve the state-of-the-art in RGB-D scene labeling. First, we present a novel combination of depth and color features to recognize different object categories in isolation. Then, we use a context model that exploits detection results of other objects in the scene to jointly optimize labels of co-occurring objects in the scene. Finally, we investigate the use of social media mining to develop the context model, and provide an investigation of its convergence. We perform thorough experimentation on both the publicly available RGB-D Dataset from the University of Washington as well as on the NYU scene dataset. An analysis of the results shows interesting insights about contextual object category recognition, and its benefits.