Estimating the uncertainty in the estimated mean area under the ROC curve of a classifier

This article considers the problem of binary classification and its assessment in a distribution-free approach. We estimate the area under the ROC curve (a more general performance metric than the error rate) of a classifier using a bootstrap-based estimator. We then use the method of the influence function to estimate the uncertainty of that estimate from the very same bootstrap samples. Monte Carlo trials show that small-sample estimates can be obtained with little bias.