Tracking continuous emotional trends of participants during affective dyadic interactions using body language and speech information

We address the problem of tracking continuous levels of a participant's activation, valence and dominance during the course of affective dyadic interactions, where participants may be speaking, listening or doing neither. To this end, we extract detailed and intuitive descriptions of each participant's body movements, posture and behavior towards his interlocutor, and speech information. We apply a Gaussian Mixture Model-based approach which computes a mapping from a set of observed audio–visual cues to an underlying emotional state. We obtain promising results for tracking trends of participants' activation and dominance values, which outperform other regression-based approaches used in the literature. Additionally, we shed light into the way expressive body language is modulated by underlying emotional states in the context of dyadic interactions.

Figure optionsDownload high-quality image (196 K)Download as PowerPoint slideHighlights
► We track continuous trends of emotional attributes during affective dyadic improvisations.
► We use detailed body language and speech information of participants.
► We obtain promising performance for activation and dominance trend tracking.
► We analyze relations between emotional states and the displayed bodily behaviors.
► We discuss the challenges of continuous data annotation by multiple people.