Model-based approach to spatial–temporal sampling of video clips for video object detection by classification

• A computational approach to build a class-specific optimal key-object model sequence is proposed.
• The approach is robust to detect video objects from a video clip with a cluttered background.
• We propose an automatic training procedure by multiple alignment with dynamic programming.
• Techniques to smartly detect video objects by classification are implemented.

For a variety of applications such as video surveillance and event annotation, the spatial–temporal boundaries between video objects are required for annotating visual content with high-level semantics. In this paper, we define spatial–temporal sampling as a unified process of extracting video objects and computing their spatial–temporal boundaries using a learnt video object model. We first provide a computational approach for learning an optimal key-object codebook sequence from a set of training video clips to characterize the semantics of the detected video objects. Then, dynamic programming with the learnt codebook sequence is used to locate the video objects with spatial–temporal boundaries in a test video clip. To verify the performance of the proposed method, a human action detection and recognition system is constructed. Experimental results show that the proposed method gives good performance on several publicly available datasets in terms of detection accuracy and recognition rate.