The spatiotemporal characteristics of elementary audiovisual speech and music processing in musically untrained subjects

Previously, the EEG technique has been used to investigate the spatiotemporal properties of audiovisual (AV) processing by taking advantage of the violation of the “additive model”, which is considered to be a very conservative approach. In the present work, we used a less conservative and novel approach than the criterion of superadditivity for estimating AV interactions. Hence, we estimated AV interaction patterns by comparing the responses to AV stimuli with the averaged responses to the unimodal visual and auditory stimuli in musically untrained subjects and by presenting syllables and piano tones coupled with flashlights. Our results suggest that the two AV objects elicited consistent interaction patterns within the time course of unisensory processing in the time range between 80 and 250 ms post stimulus onset. The scalp topographies, as well as the source estimation approach we adopted, indicate that the first interaction pattern at around 100 ms was partially driven by auditory-related cortical regions. Additionally, we found evidence for a second interaction pattern at around 200 ms that was mainly associated with the responsiveness of extra-sensory brain regions. During this later processing stage, only the music condition was associated with putative responses that originated from auditory-related cortical fields. This study provides a novel approach to investigate the basic principles underlying elementary AV speech and music processing in subjects without formal musical education.

► We investigated AV interactions related to speech and music processing.
► We found interaction effects within the time course of unisensory processing.
► AV interactions at around 100 ms were partially driven by auditory-related regions.
► AV interactions at around 200 ms were more likely related to extra-sensory brain regions.