Cohen’s kappa is a weighted average

The κκ coefficient is a popular descriptive statistic for summarizing an agreement table. It is sometimes desirable to combine some of the categories, for example, when categories are easily confused, and then calculate κκ for the collapsed table. Since the categories of an agreement table are nominal and the order in which the categories of a table are listed is irrelevant, combining categories of an agreement table is identical to partitioning the categories in subsets.In this paper we prove that given a partition type of the categories, the overall κκ-value of the original table is a weighted average of the κκ-values of the collapsed tables corresponding to all partitions of that type. The weights are the denominators of the kappas of the subtables. An immediate consequence is that Cohen’s κκ can be interpreted as a weighted average of the κκ-values of the agreement tables corresponding to all non-trivial partitions.The κκ-value of the 2×22×2 table that is obtained by combining all categories other than the one of current interest into a single “all others” category, reflects the reliability of the individual category. Since the overall κκ-value is a weighted average of these 2×22×2 κκ-values the category reliability indicates how a category contributes to the overall κκ-value. It would be good practice to report both the overall κκ-value and the category reliabilities of an agreement table.

► Cohen’s kappa is a weighted average of the kappas of all tables of a partition type. ► Kappa is a weighted average of kappas of all possible collapsed agreement tables. ► Reporting both Kappa and the category reliabilities is more useful than Kappa alone.