Facial descriptors for human interaction recognition in still images

•We explore the contribution of face features to interaction recognition in images.•Several novel facial feature descriptors are proposed.•A new dataset for human interaction recognition in still images is collected.•Scene and Convolutional Neural Network (CNN) features are also explored.•Experiments show that faces provide complementary information for recognition.

This paper presents a novel approach in a rarely studied area of computer vision: Human interaction recognition in still images. We explore whether the facial regions and their spatial configurations contribute to the recognition of interactions. In this respect, our method involves extraction of several visual features from the facial regions, as well as incorporation of scene characteristics and deep features to the recognition. Extracted multiple features are utilized within a discriminative learning framework for recognizing interactions between people. Our designed facial descriptors are based on the observation that relative positions, size and locations of the faces are likely to be important for characterizing human interactions. Since there is no available dataset in this relatively new domain, a comprehensive new dataset which includes several images of human interactions is collected. Our experimental results show that faces and scene characteristics contain important information to recognize interactions between people.