A new framework with multiple tasks for detecting and locating pain events in video

Automatically detecting and locating pain events in video is an important task in medical assessment. It is a challenging problem in facial expression analysis due to spontaneous faces, head movements and pose variations. In this paper, we explore the role of facial information at various time scales (frame, segment and sequence) and propose a new framework for pain event detection and locating in video. We introduce a feature-level fusion method for pain event detection and a multiple-task fusion method for locating pain events, respectively. Both spatial and spatial-temporal features are utilized in our study. At first, we employ the histogram of oriented gradients (HOG) of fiducial points (P-HOG) to extract spatial features from each video frame and train an SVM as a frame-based pain event detector. Secondly, HOG from Three Orthogonal Planes (named as HOG-TOP) is used to characterize the dynamic textures of a video segment, a segment-based classifier (SVM) is then trained for segment-level detection. We further apply a max pooling strategy to obtain the global P-HOG and HOG-TOP to represent the whole video sequence and a multiple kernel fusion is employed to find an optimal feature-level fusion. An SVM with multiple kernels is trained to perform sequence-level (pain event) detection. Finally, an effective probabilistic fusion method is proposed to integrate the detection results of the three different tasks (frame-level, segment-level and sequence-level detection) to locate pain events in video. Extensive experiments conducted on the UNBC-McMaster Shoulder Pain database show that our proposed method outperforms other state-of-the-art methods both in pain event detection and locating in video. Our sequence-level event detection method has also been applied to facial expression recognition in video with good results.