Probabilistic semantic models for manipulation action representation and extraction

•We present a framework for analysis of manipulation action primitives.•The semantics of actions are modeled as probabilistic distributions of semantic events over action phase.•The resulting probabilistic models are used in combination with lower level trajectories.•The approach is evaluated on a problem of extracting action primitives from observation.

In this paper we present a hierarchical framework for representation of manipulation actions and its applicability to the problem of top down action extraction from observation. The framework consists of novel probabilistic semantic models, which encode contact relations as probability distributions over the action phase. The models are action descriptive and can be used to provide probabilistic similarity scores for newly observed action sequences. The lower level of the representation consists of parametric hidden Markov models, which encode trajectory information.