The effects of visual beats on prosodic prominence: Acoustic analyses, auditory perception and visual perception

Speakers employ acoustic cues (pitch accents) to indicate that a word is important, but may also use visual cues (beat gestures, head nods, eyebrow movements) for this purpose. Even though these acoustic and visual cues are related, the exact nature of this relationship is far from well understood. We investigate whether producing a visual beat leads to changes in how acoustic prominence is realized in speech, and whether it leads to changes in how prominence is perceived by observers. For Experiment I (“making beats”) we use an original experimental paradigm in which speakers are instructed to realize a target sentence with different distributions of acoustic and visual cues for prominence. Acoustic analyses reveal that the production of a visual beat indeed has an effect on the acoustic realization of the co-occurring speech, in particular on duration and the higher formants (F2 and F3), independent of the kind of visual beat and of the presence and position of pitch accents. In Experiment II (“hearing beats”), it is found that visual beats have a significant effect on the perceived prominence of the target words. When a speaker produces a beat gesture, an eyebrow movement or a head nod, the accompanying word is produced with relatively more spoken emphasis. In Experiment III (“seeing beats”), finally, it is found that when participants see a speaker realize a visual beat on a word, they perceive it as more prominent than when they do not see the beat gesture.