Learning hierarchical spatio-temporal pattern for human activity prediction

•A novel approach to learning a hierarchical spatio-temporal pattern of human actions.•Spatio-temporal pattern can be learned by a Hierarchical Self-Organizing Map (HSOM).•The associative weights between HSOM can be obtained through Hebbian learning.•Ongoing activities can be predicted by Variable order Markov Model (VMM).

Human activity prediction has become increasingly valuable in many applications. This paper, initially from the perspective of cognition science, presents a novel approach to learning a hierarchical spatio-temporal pattern of human activities to predict ongoing activities from videos that contain only the onsets of the activities. Spatio-temporal pattern can be learned by a Hierarchical Self-Organizing Map (HSOM), which consists of two self-organizing maps (i.e., action map and actionlet map) connected via associative links trained by Hebbian learning. Ongoing activities can be predicted by Variable order Markov Model (VMM), which provides the means for capturing both large and small order Markov dependencies based on the training actionlet sequences. Experiments of the proposed method on four challenging 3D action datasets captured by commodity depth cameras show promising results.