An exact test of the accuracy of binary classification models based on the probability distribution of the average rank

We propose a new way to evaluate the discriminatory power of models that generate a continuous value as the basis for performing a binary classification task. Our hypothesis test uses the average rank of the kk successes in the sample of size nn, based on those continuous values. We derive the probability mass function for the average rank from the coefficients of a Gaussian polynomial distribution that results from randomly sampling kk distinct positive integers, all nn or less. The significance level of the test is found by counting the number of arrangements that produce average ranks more extreme than the one observed. Recursive relationships can be used to calculate the values necessary to compute the pp-value. For large values of kk and nn, for which exact computation might be prohibitive, we present numerical results which indicate that the critical values of the distribution are nearly linear in nn for a fixed kk and that the coefficients of the linear relationships are nonlinear functions of kk and the desired percentile. We develop regression models for those relationships to approximate the number of arrangements in order to make the test practical for large values of kk and nn.