Simultaneous testing of multiple hypotheses using generalized p-values

In the context of simultaneously testing many hypotheses, Benjamini and Hochberg [1995. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B (Methodological) 57, 289–300] propose a procedure that guarantees that the false discovery rate (FDR) will be less than or equal to a specified value. Here, the FDR is the expected value of the ratio of the number of incorrectly rejected hypotheses and the total number of rejected hypotheses. To our knowledge, all of the existing research assumes that a usual p-value is available for each hypothesis. However, in circumstances when nuisance parameters are present, the usual p-values may not be free of the nuisance parameters. We develop a simultaneous testing procedure to control the FDR for the problem of simultaneously testing many Behrens–Fisher problems. Our multiple testing procedure, based on generalized p-values [Tsui, K.-W., Weerahandi, S., 1989. Generalized p-values in significance testing of hypotheses in the presence of nuisance parameters. J. Amer. Statist. Assoc. 84(406), 602–607], is then illustrated with an application to data from a microarray experiment.