Multiple test functions and adjusted pp-values for test statistics with discrete distributions

•Multiple test functions and adjusted abstract randomized pp-values for multiple hypothesis testing with discrete test statistics are developed.•Computational and graphical tools for multiple test functions and adjusted pp-values are developed.•A new adjusted nonrandomized pp-value is introduced and studied.•New methods are analytically compared to traditional methods.•The proposed methodology is used to analyze data.

The randomized pp-value, (nonrandomized) mid-pp-value and abstract randomized pp-value have all been recommended for testing a null hypothesis whenever the test statistic has a discrete distribution. This paper provides a unifying framework for these approaches and extends it to the multiple testing setting. In particular, multiplicity adjusted versions of the aforementioned pp-values and multiple test functions are developed. It is demonstrated that, whenever the usual nonrandomized and randomized decisions to reject or retain the null hypothesis may differ, the (adjusted) abstract randomized pp-value and test function should be reported, especially when the number of tests is large. It is shown that the proposed approach dominates the traditional randomized and nonrandomized approaches in terms of bias and variability. Tools for plotting adjusted abstract randomized pp-values and for computing multiple test functions are developed. Examples are used to illustrate the method and to motivate a new type of multiplicity adjusted mid-pp-value.