Describing visual scenes: Towards a neurolinguistics based on construction grammar

The present paper is part of a larger effort to locate the production and perception of language within the broader context of brain mechanisms for action and perception more generally. Here we model function in terms of the competition and cooperation of schemas. We use the task of describing visual scenes to explore the suitability of Construction Grammar as an appropriate framework for a schema-based linguistics. We recall the early VISIONS model of schema-based computer analysis of static visual scenes and then introduce SemRep as a graphical representation of dynamic visual scenes designed to support the generation of varied descriptions of episodes. We report preliminary results on implementing the production of sentences using Template Construction Grammar (TCG), a new form of construction grammar distinguished by its use of SemRep to express semantics. We summarize data on neural correlates relevant to future work on TCG within the context of neurolinguistics, and show how the relation between SemRep and TCG can serve as the basis for modeling language comprehension.