Latent semantic learning with structured sparse representation for human action recognition

This paper proposes a novel latent semantic learning method for extracting high-level latent semantics from a large vocabulary of abundant mid-level features (i.e. visual keywords) with structured sparse representation, which can help to bridge the semantic gap in the challenging task of human action recognition. To discover the manifold structure of mid-level features, we develop a graph-based spectral embedding approach to latent semantic learning, with the graph over mid-level features being constructed using sparse representation. Moreover, we define new L1-normL1-norm hypergraph regularization to induce extra structured sparsity into sparse representation for graph construction. Due to the nice properties (sparsity and noise-robustness) of such structured sparse representation, our graph construction can capture dominant and robust relationships among mid-level features, which are crucial for the success of latent semantic learning in action recognition. Unlike the traditional latent semantic analysis based on topic models, our latent semantic learning method can explore the manifold structure of mid-level features in both graph construction and spectral embedding, which results in compact but discriminative high-level features. The experimental results on the commonly used KTH action dataset and unconstrained YouTube action dataset show the promising performance of our method.

► We have developed novel structured sparse coding in human action recognition. ► The manifold structure of visual keywords can be used for latent semantic learning. ► Our method has been shown to achieve promising results in action recognition.