Cohen’s kappa can always be increased and decreased by combining categories

The kappa coefficient is a popular descriptive statistic for summarizing the cross classification of two nominal variables with identical categories. It has been frequently observed in the literature that combining two categories increases the value of kappa. In this note we prove the following existence theorem for kappa: For any nontrivial k×kk×k agreement table with k∈N≥3k∈N≥3 categories, there exist two categories such that, when combined, the kappa value of the collapsed (k−1)×(k−1)(k−1)×(k−1) agreement table is higher than the original kappa value. In addition, there exist two categories such that, when combined, the kappa value of the collapsed table is smaller than the original kappa value.