Automatic extraction of relevant video shots of specific actions exploiting Web data

• A novel method to automatically extract action video shots from the Web videos.
• Large-scale experiments with 100 human actions and 12 non-human actions.
• Exploiting action images helps enhance significantly performance.
• Employing human pose matching improves results of human actions.

Video sharing websites have recently become a tremendous video source, which is easily accessible without any costs. This has encouraged researchers in the action recognition field to construct action database exploiting Web sources. However Web sources are generally too noisy to be used directly as a recognition database. Thus building action database from Web sources has required extensive human efforts on manual selection of video parts related to specified actions. In this paper, we introduce a novel method to automatically extract video shots related to given action keywords from Web videos according to their metadata and visual features. First, we select relevant videos among tagged Web videos based on the relevance between their tags and the given keyword. After segmenting selected videos into shots, we rank these shots exploiting their visual features in order to obtain shots of interest as top ranked shots. Especially, we propose to adopt Web images and human pose matching method in shot ranking step and show that this application helps to boost more relevant shots to the top. This unsupervised method of ours only requires the provision of action keywords such as “surf wave” or “bake bread” at the beginn ing. We have made large-scale experiments on various kinds of human actions as well as non-human actions and obtained promising results.