ROC curves and nonrandom data

- This paper shows that ROC curves that are constructed with nonrandom data are biased.
- The magnitude of this bias is explored using simulations.
- A procedure for plotting consistent ROC curves is introduced.
- The presented procedure works well with simulated and non-simulated data.

This paper shows that when a classifier is evaluated with nonrandom test data, ROC curves differ from the ROC curves that would be obtained with a random sample. To address this bias, this paper introduces a procedure for plotting ROC curves that are inferred from nonrandom test data. I provide simulations to illustrate the procedure as well as the magnitude of bias that is found in empirical ROC curves constructed with nonrandom test data. The paper also includes a demonstration of the procedure on (non-simulated) data used to model wine preferences in the wine industry.