Multi-label learning with missing labels for image annotation and facial action unit recognition

•An inductive method is proposed to handle missing labels in multi-label learning.•The label bias of treating missing labels as negative labels is avoided.•Label consistency, example-level and class-level smoothness are considered.•We present an efficient algorithm to learn a parametric classifier.•The proposed method is applied to image annotation and facial action recognition.

Many problems in computer vision, such as image annotation, can be formulated as multi-label learning problems. It is typically assumed that the complete label assignment for each training image is available. However, this is often not the case in practice, as many training images may only be annotated with a partial set of labels, either due to the intensive effort to obtain the fully labeled training set or the intrinsic ambiguities among the classes. In this work, we propose a method for multi-label learning that explicitly handles missing labels. We train classifiers with the multi-label with missing labels (MLML) learning framework by enforcing the consistency between the predicted labels and the provided labels as well as the local smoothness among the label assignments. Experiments on three benchmark data sets in image annotation and one benchmark data set in facial action unit recognition demonstrate the improved performance of our method in comparison of several state-of-the-art methods.