Hybrid graphical model for semantic image segmentation

• Proposed a hybrid model integrates Conditional Random Field and Bayesian Network.
• Finer and balanced segmentation for both background and foreground is adopted for higher-order statistics.
• We build a new model for contextual relationships of semantic sub-scenes.
• The BN is used to model local contextual interactions for each semantic sub-scene.
• Experimental results show that the hybrid model outperforms pure CRF models.

To make full use of both non-causal and causal cues in natural images, we propose a hybrid hierarchical Conditional Random Field (HCRF) and Bayesian Network (BN) model for semantic image segmentation in this paper. The HCRF is used to capture non-causal relationship, such as appearance features and inter-class co-occurrence statistics, to produce initial semantic sub-scene predictions. Whereas, the BN is used to model contextual interactions for each semantic sub-scene in the form of class statistics from its neighboring regions, of which its conditional probabilities are learned automatically from training data. The learned BN structure is then used to encode the structure of contextual dependencies for sub-scenes in the initial predictions to generate final refined predictions. Experiments on the Stanford 8-class dataset and the LHI 15-class dataset show that the hybrid model outperforms pure CRF models by 2–4% in average classification accuracy.