Audiovisual perception of openness and lip rounding in front vowels

Swedish nonsense syllables /ɡiɡ/, /ɡyɡ/, /ɡeɡ/ and /ɡøɡ/, produced by four speakers, were video-recorded and presented to male and female subjects in auditory, visual and audiovisual mode and also in cross-dubbed audiovisual form with incongruent cues to vowel openness, roundedness, or both. With audiovisual stimuli, subjects perceived openness nearly always by ear. Most subjects perceived roundedness by eye rather than by ear although the auditory conditions were optimal and the reported percept was an auditory one. This resulted in fusions such as when an acoustic /ɡeɡ/ dubbed onto an optic /ɡyɡ/ was predominantly perceived as /ɡøɡ/. Since the acoustic cues to openness are prominent, while those to roundedness are less reliable, this lends support to the “information reliability hypothesis” in multisensory perception: The perception of a feature is dominated by the modality that provides the more reliable information. A mostly male minority relied less on vision. The between-gender difference was significant. Presence of lip rounding (a visibly marked feature) was noticed more easily than its absence. The influence of optic information was not fully explicable on the basis of the subjects’ success rates in lipreading compared with auditory perception. It was highest in stimuli produced by a speaker who smiled.