Spatio-temporal feature-based keyframe detection from video shots using spectral clustering

Keyframe detection is a fundamental component in approaches for large-scale mapping and scene recognition. Assuming that the detection is applied to a set of continuously captured frames, this paper presents a keyframe detector that not only considers the frame content to quantify appearance changes on the sequence, but also the temporal accumulation of evidence. If frames are described as a set of local features, our algorithm proposes a unified framework for comparing local features acquired from consecutive frames by the building of an auxiliary graph-based on the locality of features. Spectral clustering is then employed to obtain tentative graph partitions. Validated partitions will be associated to keyframes. It should be noted that the approach does not need to estimate the motion of the camera, and that the similarity measure defined within this framework can be used for any sort of feature. Experimental results using different types of visual features show the strength of our representation. Moreover, an evaluation methodology has been defined for the quantitative comparison of our keyframe detector against other similar approaches.

► Keyframe detection using spectral clustering. ► Keyframe detection using an independent graph-based representation. ► A similarity measure for a sequence of images is defined. ► The graph representation allows the use of the algorithm for different sort of features or sensors.