Visual object-action recognition: Inferring object affordances from human demonstration

This paper investigates object categorization according to function, i.e., learning the affordances of objects from human demonstration. Object affordances (functionality) are inferred from observations of humans using the objects in different types of actions. The intended application is learning from demonstration, in which a robot learns to employ objects in household tasks, from observing a human performing the same tasks with the objects. We present a method for categorizing manipulated objects and human manipulation actions in context of each other. The method is able to simultaneously segment and classify human hand actions, and detect and classify the objects involved in the action. This can serve as an initial step in a learning from demonstration method. Experiments show that the contextual information improves the classification of both objects and actions.

Research highlights► The affordances of objects can notin general be deduced from object appearance only. ► We suggest a method for recognizing object affordances from human demonstration. ► This is done by simultaneous contextual categorization of objects and human actions. ► The method segment and classifies hand actions, and detect and classifies objects. ► This can serve as an initial step in a learning from demonstration method.