Social eye gaze modulates processing of speech and co-speech gesture

• We investigate the interplay of gaze, gesture and speech during comprehension.
• Speaker-gaze direction modulates recipients’ comprehension of speech and gesture.
• Recipients that are not gazed at process speech less well than gazed at addressees.
• However, perceiving gestures allowed them to process speech as well as addressees.
• When unaddressed recipients’ speech processing suffers, co-speech gestures help.

In human face-to-face communication, language comprehension is a multi-modal, situated activity. However, little is known about how we combine information from different modalities during comprehension, and how perceived communicative intentions, often signaled through visual signals, influence this process. We explored this question by simulating a multi-party communication context in which a speaker alternated her gaze between two recipients. Participants viewed speech-only or speech + gesture object-related messages when being addressed (direct gaze) or unaddressed (gaze averted to other participant). They were then asked to choose which of two object images matched the speaker’s preceding message. Unaddressed recipients responded significantly more slowly than addressees for speech-only utterances. However, perceiving the same speech accompanied by gestures sped unaddressed recipients up to a level identical to that of addressees. That is, when unaddressed recipients’ speech processing suffers, gestures can enhance the comprehension of a speaker’s message. We discuss our findings with respect to two hypotheses attempting to account for how social eye gaze may modulate multi-modal language comprehension.