Saliency-based dense trajectories for action recognition using low-rank matrix decomposition

Dense trajectory methods have recently been proved to be successful in recognizing actions in realistic videos. However, their performance is still limited due to the uniform dense sampling, which does not discriminate between action-related areas and background. This paper proposes to improve the dense trajectories for recognizing actions captured in realistic scenes, especially in the presence of camera motion. Firstly, based on the observation that the motion in action-related areas is usually much more irregular than the camera motion in background, we recover the salient regions in a video by implementing low-rank matrix decomposition on the motion information and use the saliency maps to indicate action-related areas. Considering action-related regions are changeable but continuous with time, we temporally split a video into subvideos and compute the salient regions subvideo by subvideo. In addition, to ensure spatial continuity, we spatially divide a subvideo into patches and arrange the vectorized optical flow of all the spatial patches to collect the motion information for salient region detection. Then, after the saliency maps of all subvideos in a video are obtained, we incorporate them into dense tracking to extract saliency-based dense trajectories to describe actions. To evaluate the performance of the proposed method, we conduct experiments on four benchmark datasets, namely, Hollywood2, YouTube, HMDB51 and UCF101, and show that the performance of our method is competitive with the state of the art.