Semantic video scene segmentation and transfer

•New semi-supervised method to semantically segment a scene based on video activity.•Learned functional categories are used to segment different scenes (scene transfer).•We introduce new trajectory features useful for semantic segmentation.•Only a small subset of relevant features leads to high classification accuracy.•Proposed method achieves state-of-the-art scene classification and transfer results.

In this paper we present a new approach to semantically segment a scene based on video activity and to transfer the semantic categories to other, different scenarios. In the proposed approach, a user annotates a few scenes by labeling each area with a functional category such as background, entry/exit, walking path, interest point. For each area, we calculate features derived from object tracks computed in real-time on hours of video. The characteristics of each functional area learned in the labeled training sequences are then used to classify regions in different scenarios. We demonstrate the proposed approach on several hours of three different indoor scenes, where we achieve state-of-the-art classification results.