The EEG and fMRI signatures of neural integration: An investigation of meaningful gestures and corresponding speech

One of the key features of human interpersonal communication is our ability to integrate information communicated by speech and accompanying gestures. However, it is still not fully understood how this essential combinatory process is represented in the human brain. Functional magnetic resonance imaging (fMRI) studies have unanimously attested the relevance of activation in the posterior superior temporal sulcus/middle temporal gyrus (pSTS/MTG), while electroencephalography (EEG) studies have shown oscillatory activity in specific frequency bands to be associated with multisensory integration. In the current study, we used fMRI and EEG to separately investigate the anatomical and oscillatory neural signature of integrating intrinsically meaningful gestures (IMG; e.g. “Thumbs-up gesture”) and corresponding speech (e.g., “The actor did a good job”). In both the fMRI (n=20) and EEG (n=20) study, participants were presented with videos of an actor either: performing IMG in the context of a German sentence (GG), IMG in the context of a Russian (as a foreign language) sentence (GR), or speaking an isolated German sentence without gesture (SG). The results of the fMRI experiment confirmed that gesture-speech processing of IMG activates the posterior MTG (GG>GR∩GG>SG). In the EEG experiment we found that the identical integration process (GG>GR∩GG>SG) is related to a centrally-distributed alpha (7-13 Hz) power decrease within 700-1400 ms post-onset of the critical word. These new findings suggest that BOLD response increase in the pMTG and alpha power decrease represent the neural correlates of integrating intrinsically meaningful gestures with their corresponding speech.