Bootstrap hypothesis testing for some common statistical problems: A critical evaluation of size and power properties

The construction of bootstrap hypothesis tests can differ from that of bootstrap confidence intervals because of the need to generate the bootstrap distribution of test statistics under a specific null hypothesis. Similarly, bootstrap power calculations rely on resampling being carried out under specific alternatives. We describe and develop null and alternative resampling schemes for common scenarios, constructing bootstrap tests for the correlation coefficient, variance, and regression/ANOVA models. Bootstrap power calculations for these scenarios are described. In some cases, null-resampling bootstrap tests are equivalent to tests based on appropriately constructed bootstrap confidence intervals. In other cases, particularly those for which simple percentile-method bootstrap intervals are in routine use such as the correlation coefficient, null-resampling tests differ from interval-based tests. We critically assess the performance of bootstrap tests, examining size and power properties of the tests numerically using both real and simulated data. Where they differ from tests based on bootstrap confidence intervals, null-resampling tests have reasonable size properties, outperforming tests based on bootstrapping without regard to the null hypothesis. The bootstrap tests also have reasonable power properties.