Multi-task human action recognition via exploring super-category

•An novel action recognition approach is proposed based on MTL framework with super-category.•Super-category is explored by measuring the similarity among action categories.•Feature sharing and competition are encouraged simultaneously with super-category.

There is indeed a relationship among various action categories, with which many correlated action categories can be clustered into a same group, named super-category. Knowledge sharing within super-category is an effective strategy to achieve good generalization performance. In this paper, we propose a novel human action recognition method based on multi-task learning framework with super-category. We employ Fisher vector as the action representation by concatenating the gradients of log likelihood with respect to mean vector and covariance parameters of Gaussion Mixture Model. Considering the occupancy probability of each Gaussian component is different, we naturally discover the relationship among different action categories by evaluating the importance of each Gaussian component in classifying each category. For these categories, the more related to the same Gaussian component, the more possible belonging to the same super-category, and vice versa. By applying the explored super-category information as a prior, feature sharing within super-category and feature competition between super-categories are simultaneously encouraged in multi-task learning framework. Experimental results on large and realistic datasets HMDB51 and UCF50 show that the proposed method achieves higher accuracy with less dimensions of features over several state-of-the-art approaches.