Quality control of crowdsourced classification using hierarchical class structures

• We proposed a quality control method for crowdsourced hierarchical classification.
• Our method captures worker abilities by incorporating hierarchical class structures.
• The experiments using real crowdsourced datasets showed that our method is effective.
• Our method is helpful to multi-class classification with an implicit hierarchy.

Crowdsourcing is an emerging approach to utilize a large pool of human workers and execute various intelligent tasks. Repeated labeling is a widely adopted quality control method in crowdsourcing. This method is based on selecting one reliable label from multiple labels collected by workers because a single label from only one worker has a wide variance of accuracy. Hierarchical classification, where each class has a hierarchical relationship, is a typical task in crowdsourcing and used to organize information in many knowledge systems. However, direct applications of existing methods designed for multi-class classification have the disadvantage of discriminating among a large number of classes. In this paper, we propose a label aggregation method for hierarchical classification tasks. Our method takes the hierarchical structure into account to handle a large number of classes and estimate worker abilities more precisely.Our method is inspired by the steps model based on item response theory, which models responses of examinees to sequentially dependent questions. We considered the hierarchical classification to be a question consisting of a sequence of sub-questions and built a worker response model for hierarchical classification. We conducted experiments using real crowdsourced hierarchical classification tasks for book classification and business classification and demonstrated the benefit of incorporating a hierarchical structure to improve the label aggregation accuracy. Our method also improves the accuracy for multi-class classification task for adult content classification with an implicit hierarchical structure among classes.