Active learning with uncertain labeling knowledge

• The oracle is regarded to have uncertain labeling knowledge.
• An objective function is defined to actively select instance and oracle.
• We use the diversity density framework to characterize the uncertain knowledge.
• We employ an error-reduction-based mechanism to verify the labeling information.

Traditional active learning assumes that the labeler is capable of providing ground truth label for each queried instance. In reality, a labeler might not have sufficient knowledge to label a queried instance but can only guess the label with his/her best knowledge. As a result, the label provided by the labeler, who is regarded to have uncertain labeling knowledge, might be incorrect. In this paper, we formulate this problem as a new “uncertain labeling knowledge” based active learning paradigm, and our key is to characterize the knowledge set of each labeler for active learning. By taking each unlabeled instance’s information and its likelihood of belonging to the uncertain knowledge set as a whole, we define an objective function to ensure that each queried instance is the most informative one for labeling and the labeler should also have sufficient knowledge to label the instance. To ensure label quality, we propose to use diversity density to characterize a labeler’s uncertain knowledge and further employ an error-reduction-based mechanism to either accept or decline a labeler’s label on uncertain instances. Experiments demonstrate the effectiveness of the proposed algorithm for real-world active learning tasks with uncertain labeling knowledge.