Table-top scene analysis using knowledge-supervised MCMC

•This approach generates abstract scene graphs from uncertain 6D pose estimates.•The proposed system is realized by the knowledge-supervised MCMC sampling technique.•Task-specific context knowledge is defined as descriptive rules in Markov logic networks.•This approach links the high-level abstract scene description to uncertain low level measurements.•False estimates and hidden objects are systematically inferred using the defined knowledge base.

In this paper, we propose a probabilistic approach to generate abstract scene graphs from uncertain 6D pose estimates. We focus on generating a semantic understanding of the perceived scenes that well explains the composition of the scene and the inter-object relations. The proposed system is realized by our knowledge-supervised MCMC sampling technique. We explicitly make use of task-specific context knowledge by encoding this knowledge as descriptive rules in Markov logic networks. We use a probabilistic sensor model to encode the fact that measurements are subject to significant uncertainty. We integrate the measurements with the abstract scene graph in a data driven MCMC process. Our system is fully probabilistic and links the high-level abstract scene description to uncertain low level measurements. Moreover, false estimates of the object poses and hidden objects of the perceived scenes can be systematically detected using the defined Markov logic knowledge base. The effectiveness of our approach is demonstrated and evaluated in real world experiments.