Model-free incremental learning of the semantics of manipulation actions

•We addressed the problem of on-line learning of the semantics of manipulations.•This is the first attempt to apply reasoning at the semantic level for learning.•Our framework is fully grounded at the signal level.•We introduced a new benchmark with 8 manipulations including in total 120 samples.•We evaluated the learned semantic models with 20 long manipulation sequences.

Understanding and learning the semantics of complex manipulation actions are intriguing and non-trivial issues for the development of autonomous robots. In this paper, we present a novel method for an on-line, incremental learning of the semantics of manipulation actions by observation. Recently, we had introduced the Semantic Event Chains (SECs) as a new generic representation for manipulations, which can be directly computed from a stream of images and is based on the changes in the relationships between objects involved in a manipulation. We here show that the SEC concept can be used to bootstrap the learning of the semantics of manipulation actions without using any prior knowledge about actions or objects. We create a new manipulation action benchmark with 8 different manipulation tasks including in total 120 samples to learn an archetypal SEC model for each manipulation action. We then evaluate the learned SEC models with 20 long and complex chained manipulation sequences including in total 103 manipulation samples. Thereby we put the event chains to a decisive test asking how powerful is action classification when using this framework. We find that we reach up to 100%100% and 87%87% average precision and recall values in the validation phase and 99%99% and 92%92% in the testing phase. This supports the notion that SECs are a useful tool for classifying manipulation actions in a fully automatic way.