Unsupervised video categorization based on multivariate information bottleneck method

• Novel multivariate IB model is proposed for unsupervised video categorization.
• Effective solution is designed to integrate multiple features simultaneously.
• Information-theoretic optimization is constructed to alleviate the semantic gap.

The integration of multiple features is important for action categorization and object recognition in videos, because single feature based representation hardly captures imaging variations and individual attributes. In this paper, a novel formulation named Multivariate video Information Bottleneck (MvIB) is defined. It is an extensional type of multivariate information bottleneck and can discover categories from a collection of unlabeled videos automatically. Differing from the original multivariate information bottleneck, the novel approach extracts the video categories from multiple features simultaneously, such as local static and dynamic feature, each type of feature is treated as a relevant variable. Specifically, by preserving the relevant information with respect to these feature variables maximally, the MvIB method is able to integrate various aspects of semantic information into the final video partitioning results, and thus captures the complementary information resided in multiple feature variables. Extensive experimental results on five challenging video data sets show that the proposed approach can consistently and significantly outperform other state-of-the-art unsupervised learning methods.