Handheld object detection and its related event analysis using ratio histogram and mixture of HMMs

•A novel scheme is proposed for detecting handheld objects from videos.•A coding technique is proposed for converting frames to various action primitives.•A new multiplicity concept is addressed to represent an event more accurately.•A novel method is proposed to capture the dynamics of an event.

This paper proposes a novel system to analyze human-object interaction events happening between hands and faces in real time. Two challenging problems in this event analysis must be addressed, i.e., there is no prior knowledge (like shape, color, size, and texture) about the handheld objects, and there are large spatial–temporal variations in event representation. For the first challenge, a novel ratio histogram is proposed to find important color bins to locate handheld objects and their trajectories via a code book technique. This scheme is different from other boosted methods which require very time-consuming estimations to search reliable body configurations. For the second challenge, a mixture of HMMs is proposed to describe an event not only from its dynamic context but also its multiplicity context. It can be performed in real time because an exhaustive search process is avoided to find possible interaction pairs between objects and body parts.