A probabilistic model of active learning with multiple noisy oracles

In this paper, we focus on obtaining an accurate classifier in active learning, when there are multiple noisy oracles with different and unknown levels of expertise to provide labels for selected instances. We propose a probabilistic model of active learning with multiple noisy oracles (PMActive). Our goal is formulized as to select the most reliable oracle and estimate the actual label on training data. When an instance is selected in every round of active learning, we firstly model the accuracies of individual oracles based on observed noisy labels, and select the most reliable oracle of all to provide a label for the instance. After adding the new instance-label pair into the training set, the actual label of the instance is estimated and used for enhancing the performance of the current classifier. The experimental results indicate that the PMActive method can work with different noise levels of oracles. Compared with the baselines which are commonly used in this area of active learning, the PMActive method is superior in obtaining a more accurate classifier.