Exploring the contribution of prosody and gesture to the perception of focus using an animated agent

Speech prosody has traditionally been analyzed in terms of acoustic features. Although visual features have been shown to enhance linguistic processing, the conventional view is that facial and body gesture information in oral (non-signed) languages tends to be redundant and has the role of helping the hearer recover the meaning of an utterance. Though prosodic information in face-to-face communication is produced with concurrent visual information, little is known about their audiovisual multisensory interactions. We conducted two perception experiments modeled after the McGurk paradigm with a 3D animated character, in which varying degrees of discordance between auditory and visual information were created to investigate two related questions regarding the detection of contrastive-corrective focus: (a) how important are gestural cues with respect to auditory cues and (b) what is the relevance of the different gestural movements involved (i.e., head nodding, eyebrow raising)? Participants were presented with combinations of auditory and visual cues for both information and Contrastive Focus Statements (Experiment 1, with the corresponding unimodal control experiments) or combinations of two visual cues (namely combinations of competing eyebrow and head movements) without auditory information (Experiment 2), and were asked to identify whether the utterance presented was a statement or a correction. Results of Experiment 1 showed that (a) the presence of either acoustic or gestural features of contrastive focus were key in guiding the listener towards one interpretation or another, and (b) listeners were more sensitive to one of the modalities when the other was weaker. Results of Experiment 2 showed that (a) both types of visual cues (head and eyebrow movements) contributed individually to the perception of contrastive focus, and (b) head nods were more informative than eyebrow movements for focus identification. Overall, our findings suggest that prosodic and visual information work in a complementary fashion and are not integrated in the same way as auditory and visual information during segmental perception.