The influence of matching degrees of synchronous auditory and visual information in videos of real-world events on cognitive integration: an event-related potential study

In this article, we aim to study the influence of matching degrees of synchronous natural auditory and visual information on cognitive integration. Videos with matched, moderately matched, and mismatched audio-visual information were used as stimuli. The results showed that videos with moderately matched audio-visual information could elicit N400, P600, and late negativity (LN) effects, while videos with mismatched audio-visual information could elicit N400 and late negativity effects as compared with those with matched audio-visual information. It was further proven that N400 might reflect the connection process during multisensory integration, and P600 was more related to the evaluation process on the matching degrees of the audio-visual information in videos. Late negativity under the mismatched condition might be the combination of late frontal negativity (LFN) and late posterior negativity (LPN), which reflected the attention reallocating process and the recognition process, while late negativity under the moderately matched condition might be the LPN, which was related to the recognition process in the human brain. It was demonstrated that cognitive integration of synchronous audio-visual information would be modulated by different matching degrees of audio-visual information as indexed by different event-related potential (ERP) effects.

▶Videos with moderately matched audio-visual stimuli elicited N400-P600-late negativity (LN) effects. ▶Videos with mismatched audio-visual stimuli elicited N400-LN effects. ▶N400 might reflect the connection process during multisensory integration. ▶P600 was related to the evaluation process on matching degree of audio-visual stimuli. ▶Cognitive integration of audio-visual information was modulated by matching degrees.