A hierarchical positive and negative emotion understanding system based on integrated analysis of visual and brain signals

This paper proposes a novel emotion understanding system based on human brain activity, facial expression, and “GIST” to categorize emotions reflected by natural scenes, and to share emotions between humans and the developed system. Three modalities are adopted in the proposed system, including an electroencephalography (EEG) signal, human facial expressions stimulated by a natural scene, and “GIST”, which is used to extract low-level features and represent the primitive emotional gist of the scene. Through integrated analyzing the EEG of humans with different facial expressions and visual information from natural scene images, the proposed system can create clustering for different emotional features and construct some primitive knowledge about emotions. Using an interactive process through time, the proposed system can infer a human’s emotional status based on a neuro-fuzzy inference model and obtain a mental ability to understand emotions more accurately. The mean opinion score (MOS) is used to evaluate the performance of the proposed emotion understanding system. Experimental results demonstrate that an understanding of positive and negative emotions can be achieved under human supervision.