Recognizing human group action by layered model with multiple cues

•A layered model properly represents human group action at diverse granularities.•Flexibly to consider multiple cues in different levels with uniform features.•Gaussian Process based trajectory feature depicts the uncertainty in group action.•New informative visual appearance descriptions for group action are proposed.

Human actions are important contents which are helpful for video analysis and interpretation. Recently, notable methods have been proposed to recognize individual actions and pair׳s interactions, whereas recognizing more complex actions involving multiple persons remains a challenge. In this paper, we focus on the actions performed by a small group that consists of countable persons who generally act with correlative purposes. To cope with the varying number of participants and the inherent interactions within the group, we propose a layered model to describe the discriminative characteristics at different granularities and present each layer with uniform statistical representation. Depending on this model, we can flexibly represent group actions with arbitrary features at different action scales. Gaussian processes are employed to represent motion trajectories from a probabilistic perspective to handle the variability of movements within the group. Moreover, we take discriminative appearance information into account and depict participants’ visual “style” features and group׳s “shape” characters. Taking advantage of multiple cues from different levels, our approach can better represent group actions and improve the recognition accuracy. Experiments on two human group action datasets demonstrate the validity of our approach, as we achieve the state-of-the-art performance on NUS-HGA dataset and satisfactory results on Behave dataset.