ROC curve equivalence using the Kolmogorov–Smirnov test

This paper describes a simple, non-parametric and generic test of the equivalence of receiver operating characteristic (ROC) curves based on a modified Kolmogorov–Smirnov (KS) test. The test is described in relation to the commonly used techniques such as the area under the ROC curve (AUC) and the Neyman–Pearson method. We first review how the KS test is used to test the null hypotheses that the class labels predicted by a classifier are no better than random. We then propose an interval mapping technique that allows us to use two KS tests to test the null hypothesis that two classifiers have ROC curves that are equivalent. We demonstrate that this test discriminates different ROC curves both when one curve dominates another and when the curves cross and so are not discriminated by AUC. The interval mapping technique is then used to demonstrate that, although AUC has its limitations, it can be a model-independent and coherent measure of classifier performance.

► Describes a modified KS test for the equivalence of ROC curves. ► Argues AUC is a necessary but not sufficient condition for ROC equivalence. ► Illustrates efficacy of proposed test on non-dominating ROC curves. ► Demonstrates that AUC is a coherent measure of classifier performance.