Online human action recognition based on incremental learning of weighted covariance descriptors

Different from traditional action recognition based on video segments, online action recognition aims to recognize actions from an unsegmented stream of data in a continuous manner. One approach to online recognition is based on accumulation of evidence over time. This paper presents an effective framework of such an approach to online action recognition from a stream of noisy skeleton data, using a weighted covariance descriptor as a means to accumulate information. In particular, a fast incremental updating method for the weighted covariance descriptor is developed. The weighted covariance descriptor takes the following principles into consideration: past frames have less contribution to the accumulated evidence and recent and informative frames such as key frames contribute more. To determine the discriminativeness of each frame, an effective pseudo-neutral pose is proposed to recover the neutral pose from an arbitrary pose in a frame. Two recognition methods are developed using the weighted covariance descriptor. The first method applies nearest neighbor search in a set of trained actions using a Riemannian metric of covariance matrices. The second method uses a Log-Euclidean kernel based SVM. Extensive experiments on MSRC-12 Kinect Gesture dataset, Online RGBD Action dataset, and our newly collected online action recognition dataset have demonstrated the efficacy of the proposed framework in terms of latency, miss rate and error rate.