The effect of combining categories on Bennett, Alpert and Goldstein’s SS

Cohen’s kappa is the most widely used descriptive measure of interrater agreement on a nominal scale. A measure that has repeatedly been proposed in the literature as an alternative to Cohen’s kappa is Bennett, Alpert and Goldstein’s SS. The latter measure is equivalent to Janson and Vegelius’ CC and Brennan and Prediger’s kappankappan. An agreement table can be collapsed into a table of smaller size by partitioning categories into subsets. The paper presents several results on how the overall SS-value is related to the SS-values of the collapsed tables.It is shown that, if the categories are partitioned into subsets of the same size and if we consider all collapsed tables of this partition type, then the overall SS-value is equivalent to the average SS-value of the collapsed tables. This result illustrates that there are types of partitioning the categories that, on average, do not result in loss of information in terms of the SS-value. In addition, it is proved that for all other partition types, the overall SS-value is strictly smaller than the average SS-value of the collapsed tables. A consequence is that there is always at least one way to combine categories such that the SS-value increases. The SS-value increases if we combine categories on which there exists considerable disagreement.