On comparing the accuracy of competing tests of the same hypotheses from simulation data

Statistical researchers often want to compare the operating characteristics of alternative test statistics based on simulation data. This paper points out the severe limitations of current practice and points the way towards valid and more systematic methods. The main themes are that (i) accuracy measures should not depend on a specific nominal size such as 5%, (ii) powers should only be compared for tests of equal size and that receiver operating characteristic curves provide a systematic method of doing this, (iii) in comparing accuracy measures for competing tests, it is essential to take into account their typically high correlation. This is most simply and generally achieved via classical bootstrap, and (iv) relating test accuracy to application conditions is a standard regression problem to which existing technology should be applied.