When is the area under the receiver operating characteristic curve an appropriate measure of classifier performance?

The area under the receiver operating characteristic curve is a widely used measure of the performance of classification rules. This paper shows that when classifications are based solely on data describing individual objects to be classified, the area under the receiver operating characteristic curve is an incoherent measure of performance, in the sense that the measure itself depends on the classifier being measured. It significantly extends earlier work by showing that this incoherence is not a consequence of a cost-based interpretation of misclassifications, but is a fundamental property of the area under the curve itself. The paper also shows that if additional information, such as the class assignments of other objects, is taken into account when making a classification, then the area under the curve is a coherent measure, although in those circumstances it makes an assumption which is seldom if ever appropriate.

► Provides numerical evidence of the importance of the area under the ROC curve (AUC).
► Identifies situations in which the class assignments of objects are not independent.
► Shows that only in such cases is the AUC coherent, otherwise it is incoherent.
► This latter incoherence is shown to be fundamental, and independent of costs.