A family of multi-rater kappas that can always be increased and decreased by combining categories

Cohen’s kappa is a popular descriptive statistic for measuring agreement between two raters on a nominal scale. Various authors have generalized Cohen’s kappa to the case of m≥2m≥2 raters. We consider a family of multi-rater kappas that are based on the concept of gg-agreement (g=2,3,…,mg=2,3,…,m), which refers to the situation in which it is decided that there is agreement if gg out of mm raters assign an object to the same category. For the family of multi-rater kappas we prove the following existence theorem: In the case of three or more categories there exists for each multi-rater kappa κ(m,g)κ(m,g) two categories such that, when combined, the κ(m,g)κ(m,g) value increases. In addition, there exist two categories such that, when combined, the κ(m,g)κ(m,g) value decreases.