A classification performance measure considering the degree of classification difficulty

In the field of classification, classification difficulty of instances is one of vital factors that influence the performance of classifiers, however it has been totally neglected. In this paper, a new performance measure for classification algorithms based on Receiver Operator Characteristic (ROC) curves is proposed with the ability of incorporating the information of difficulty. First, a new ROC curve with the information on classification difficulty is defined, which is abbreviated as diROC curve. The curve is constructed in a two-dimensional graph, on which weighted true positive rate is plotted on Y-axis and weighted false positive rate is plotted on X-axis. The weights of true positive rates are proportional to classification difficulty index, while those of false positive rates are inversely proportional to classification difficulty index. Then, the Area Under diROC Curves, or simply diAUC, is defined to represent the performance of classifiers quantitatively. We test the diROC curves and diAUC on real-word datasets, the experimental results suggest that they are insensitive to changes in class distribution, and superior to traditional ROC curves and AUC in terms of discrimination.