Robust sequence alignment for actor–object interaction recognition: Discovering actor–object states

In this paper, we address the problem of recognizing atomic human–object interactions from videos. Our method is based on the observation that, at the moment of physical contact with the object, both the motion and appearance (i.e., shape) of the interacting person are constrained by the target object. We introduce the concept of actor–object states as the instantaneous configuration of actor and object that usually corresponds to the moment of physical contact. We argue that the information content in frames belonging to the actor–object states is descriptive of the specific interaction. We use the actor–object state concept to propose an approach in which human–object interactions are represented by a combination of image patches and velocity information extracted along tracked body-point trajectories. However, determining the set of video frames corresponding to actor–object states is challenging as, before and after physical contact, human motion and appearance may vary significantly for the same interaction type. We address this issue by means of a robust sequence-matching algorithm that discovers actor–object states by matching pairs of misaligned sequences of features. We then show how these discovered actor–object states can be used for the recognition of basic interactions with objects. Finally, we evaluate the proposed concept on classification tasks performed on a new dataset of atomic human–object interactions.

Research highlights
► A novel representation paradigm for human–object interactions in video.
► Sequence matching approach that is robust in the presence of noise.
► A pattern-subsequence discovery method that allows to recognize interactions.