Shape group Boltzmann machine for simultaneous object segmentation and action classification

Traditionally, object segmentation and action classification in vision have been solved as two independent problems, even though it is often the case that the solution to one impacts the solution to the other. In this paper, a general variational framework is developed to use shape group Boltzmann machine (SGBM) to simultaneously segment object and classify action from multiple images. Considering the foreground similarity and the background consistency in low-level images, shape group is adopted to link top-down object segmentation with bottom-up action inference. The SGBM uses deep Boltzmann machine to model the hierarchical architecture of shape group. The obtained shape representation and similarity transformation can be exploited to guide low-level image segmentation. The collaboration between high-level and low-level can efficiently improve the results of object segmentation and action classification simultaneously. Experiments performed under low-quality conditions reveal that the method can segment the test objects accurately even for severe noises, occlusions, deformations, etc.