Neural correlates of multimodal metaphor comprehension: Evidence from event-related potentials and time-frequency decompositions

•Multimodal metaphor comprehension involves image-based semantic categorization.•Theta activity indexes retrieval of semantic information process underlying the N300.•Theta activity indexes the semantic integration process underlying the N400.•Delta activity indexes the additional integration process underlying the P600.•More efforts are devoted to the semantic integration of appropriate metaphors.

The present study examined the event-related potential (ERP) and time-frequency components correlates with the comprehension process of multimodal metaphors that were represented by the combination of “a vehicle picture + a written word of an animal”. Electroencephalogram data were recorded when participants decided whether the metaphor using an animal word for the vehicle rendered by a picture was appropriate or not. There were two conditions: appropriateness (e.g., sport utility vehicles + tiger) vs. inappropriateness (e.g., sport utility vehicles + cat). The ERP results showed that inappropriate metaphor elicited larger N300 (280-360 ms) and N400 (380-460 ms) amplitude than appropriate one, which were different from previous exclusively verbal metaphor studies that rarely observed the N300 effect. A P600 (550-750 ms) was also observed and larger in appropriate metaphor condition. Besides, the time-frequency principal component analysis revealed that two independent theta activities indexed the separable processes (retrieval of semantic features and semantic integration) underlying the N300 and N400. Delta band was also induced within a later time window and best characterized the integration process underlying P600. These results indicate the specific cognitive mechanism of multimodal metaphor comprehension that is different from verbal metaphor processing, mirrored by several separable processes indexed by ERP components and time-frequency components. The present study extends the metaphor research by uncovering the functional roles of delta and theta as well as their unique contributions to the ERP components during multimodal metaphor comprehension.