A cognitive neuroscience perspective on embodied language for human–robot cooperation

This article addresses issues in embodied sentence processing from a “cognitive neural systems” approach that combines analysis of the behavior in question, analysis of the known neurophysiological bases of this behavior, and the synthesis of a neuro-computational model of embodied sentence processing that can be applied to and tested in the context of human–robot cooperative interaction. We propose a Hybrid Comprehension Model that links compact propositional representations of sentences and discourse with their temporal unfolding in situated simulations, under the control of grammar. The starting point is a model of grammatical construction processing which specifies the neural mechanisms by which language is a structured inventory of mappings from sentence to meaning. This model is then “embodied” in a perceptual-motor system (robot) which allows it access to sentence-perceptual representation pairs, and interaction with the world providing the basis for language acquisition.We then introduce a “simulation” capability, such that the robot has an internal representation of its interaction with the world. The control of this simulator and the associated representations present a number of interesting “neuro-technical” issues. First, the “simulator” has been liberated from real-time. It can run without being connected to current sensory motor experience. Second, “simulations” appear to be represented at different levels of detail. Our paper provides a framework for beginning to address the questions: how does language and its grammar control these aspects of simulation, what are the neurophysiological bases, and how can this be demonstrated in an artificial yet embodied cognitive system.