Learning classifiers from dual annotation ambiguity via a min–max framework

• We model two sources of labeling ambiguity effectively to construct a classifier.
• We modify the hinge loss to estimate true labels from multiple labelers.
• We extend the modified hinge loss to samples that are bags of multiple instances.
• The empirical results for the developed algorithms demonstrate their effectiveness.

Many pattern recognition problems confront two sources of annotation ambiguity where (1) multiple annotators have provided their versions of a class label which may not be consistent with one another, which forms multi-labeler learning; (2) and meanwhile a class label is associated with a bag of input vectors or instances rather than each individual instance and a bag is positive for a class label as long as one of its instances shows an evidence of that class, which is often referred to as multi-instance learning. Existing methods for multi-labeler learning and multi-instance learning only address one source of the labeling ambiguity. They are not trivially feasible to tackle the dual ambiguity problem. We hence propose a novel optimization framework by modifying the hinge loss to employ the weighted consensus of different labelers׳ labels and further generalizing the notion of loss functions to bags of multiple instances. The proposed formulation can be approximately solved by two mathematically tractable models that accommodate two types of labeling bias. An alternating optimization algorithm has been derived to efficiently solve the two models. The proposed algorithms outperform existing methods on benchmark data sets collected for document classification, real-life crowd-sourced data sets, and a medical problem of heart wall motion analysis with diagnoses from multiple radiologists.