Comparison of classification accuracy using Cohen’s Weighted Kappa

Many expert systems solve classification problems. While comparing the accuracy of such classifiers, the cost of error must frequently be taken into account. In such cost-sensitive applications just using the percentage of misses as the sole meter for accuracy can be misleading. Typical examples of such problems are medical and military applications, as well as data sets with ordinal (i.e., ordered) class.A new methodology is proposed here for assessing classifiers accuracy. The approach taken is based on Cohen’s Kappa statistic. It compensates for classifications that may be due to chance. The use of Kappa is proposed as a standard meter for measuring the accuracy of all multi-valued classification problems. The use of Weighted Kappa enables to effectively deal with cost-sensitive classification. When the cost of error is unknown and can only be roughly estimated, the use of sensitivity analysis with Weighted Kappa is highly recommended.