Research ReportCrossmodal interaction of facial and vocal person identity information: An event-related potential study

Hearing a voice and seeing a face are essential parts of person identification and social interaction. It has been suggested that both types of information do not only interact at late processing stages but rather interact at the level of perceptual encoding (< 200 ms). The present study analysed when visual and auditory representations of person identity modulate the processing of voices. In unimodal trials, two successive voices (S1-S2) of the same or of two different speakers were presented. In the crossmodal condition, the S1 consisted of the face of the same or a different person with respect to the following voice stimulus. Participants had to decide whether the voice probe (S2) was from an elderly or a young person. Reaction times to the S2 were shorter when these stimuli were person-congruent, both in the uni- and crossmodal conditions. ERPs recorded to the person-incongruent as compared to the person-congruent trials (S2) were enhanced at early (100-140 ms) and later processing stages (270-530 ms) in the crossmodal condition. A similar later negative ERP effect (270-530 ms) was found in the unimodal condition as well. These results suggest that identity information conveyed by a face is capable to modulate the sensory processing of voice stimuli.

Research highlights► Early interaction between vocal and facial information (< 200 ms), ► structural enconding of person-relevant features, ► later person-identification processes (> 270 ms), ► person identity node.