Dependent judgment analysis: A Markov chain based approach for aggregating crowdsourced opinions

Annotation of large-scale datasets can promisingly be done by crowd workers in a time and cost effective way. A major challenge in this area is how we aggregate the opinions received from multiple workers to derive the final judgment. Most of the crowd opinion aggregation models known so far deal with independent opinions, where the crowd workers provide their opinions unanimously and these are not visible to everyone. In real life, there are applications where an annotator can see others' opinions. This incurs a higher chance of getting biased by the other opinions. This paper addresses a new problem, hereafter termed as dependent judgment analysis, and proposes a method to derive the final judgment from a given set of independent and dependent opinions. Here, a Markov chain based aggregation method is used to handle the opinions of the crowd workers for finding a consensus. We study the performance of the proposed method on a synthetic dataset and another real-life dataset published in recent times. The proposed method is applied on these two datasets to find out the aggregated judgment. The efficacy of our proposed method is shown by comparing it with majority voting.