Task-dependent multi-task multiple kernel learning for facial action unit detection

Facial action unit (AU) detection from images and videos is a challenging research topic and has attracted great attention in the past few years. This paper presents a novel method, task-dependent multi-task multiple kernel learning (TD-MTMKL), to jointly detect the absence and presence of multiple AUs. TD-MTMKL attempts to learn an optimal kernel combination from a given set of basis kernels for each involved AU and obtain a finer depiction of AU relations through kernel combination weights. In other words, AU detection is solved as a multi-task multiple kernel learning problem, where AU relations are encoded via their SVM discriminative hyperplanes and kernel combination weights. The kernel learning increases the discriminant power of the classifier by fusing different types of facial feature representations with multiple kernels. Specifically, based on the TD-MTMKL method proposed in this paper, co-occurrence AUs share the same kernel weights while AUs with weak co-occurrence relations may employ distinct sets of kernels. Such “task-dependent” kernel learning framework seeks a trade-off between capturing commonalities and adapting to variations in modeling AU relations. Our experiments on the CK+ and DISFA databases show that our method achieved encouraging detection results of both post and spontaneous AUs compared to the state-of-the-art methods.