A study of terminology auditors' performance for UMLS semantic type assignments

► Performance of auditors on type assignments to complex UMLS concepts is studied. ► The results indicate that individual auditors are not reliable. ► The reliability of a majority opinion computed from multiple auditors is evaluated. ► The results indicate that the majority opinion is reliable. ► It is significantly more reliable than the average performance of any auditor.