Context and locality constrained linear coding for human action recognition

•We model a novel and effective human action recognition scheme.•We propose context and locality constrained linear coding method to fully describe each local feature.•“Decisive feature” and “indecisive feature” are aided to collect the contextual information.•Group-wise sparse representation based classification method is introduced to category the query samples.•Our encoding problem has both analytical solution and approximation solution.

Bag of Words (BOW) method with spatio-temporal local features has achieved great performance in human action recognition. However, most of the existing BOW approaches based on vector quantization (VQ) neglect the contextual information of each descriptor, and suffer serious quantization error. There are two main reasons for these: in the first, each local feature is only assigned to one label and second, the information about the spatial layout of the features is disregarded. In this paper, we present a novel and effective coding method called context and locality constrained linear coding (CLLC) to overcome these limitations, in which the relationships among local features and their structural information are preserved. After that, a group-wise sparse representation based classification (GSRC) method is implemented to assign the query sample into one category which yields the smallest reconstruction error. Our method is verified on the challenging databases and achieves comparable performance with state-of-the-art methods.