Coupled hidden conditional random fields for RGB-D human action recognition

• We propose cHCRF to learn sequence-specific and sequence-shared temporal structure.
• We contribute a novel RGB-D human action dataset containing 1200 samples.
• Experiments on 3 popular datasets show the superiority of the proposed method.

This paper proposes a human action recognition method via coupled hidden conditional random fields model by fusing both RGB and depth sequential information. The coupled hidden conditional random fields model extends the standard hidden-state conditional random fields model only with one chain-structure sequential observation to multiple chain-structure sequential observations, which are synchronized sequence data captured in multiple modalities. For model formulation, we propose the specific graph structure for the interaction among multiple modalities and design the corresponding potential functions. Then we propose the model learning and inference methods to discover the latent correlation between RGB and depth data as well as model temporal context within individual modality. The extensive experiments show that the proposed model can boost the performance of human action recognition by taking advance of complementary characteristics from both RGB and depth modalities.