ROC curves and nonrandom data

•This paper shows that ROC curves that are constructed with nonrandom data are biased.•The magnitude of this bias is explored using simulations.•A procedure for plotting consistent ROC curves is introduced.•The presented procedure works well with simulated and non-simulated data.

This paper shows that when a classifier is evaluated with nonrandom test data, ROC curves differ from the ROC curves that would be obtained with a random sample. To address this bias, this paper introduces a procedure for plotting ROC curves that are inferred from nonrandom test data. I provide simulations to illustrate the procedure as well as the magnitude of bias that is found in empirical ROC curves constructed with nonrandom test data. The paper also includes a demonstration of the procedure on (non-simulated) data used to model wine preferences in the wine industry.