Randomization tests to identify significant effects in experimental designs for robustness testing

The screening designs applied in robustness tests are usually fractional factorial or Plackett–Burman designs. Different methods to identify significant factor effects estimated from experimental designs for robustness testing are described. In this paper, the use of randomization tests as a statistical interpretation method is examined and compared with both graphical (half-normal probability plot) and statistical methods, such as the estimation of error based on a priori considered negligible effects and the algorithm of Dong. It was found that all statistical methods usually gave similar results, i.e. the same effects are found to be significant. However, sometimes randomization tests indicate either less or more significant factor effects compared to the other methods, regardless the design size. Both randomization tests and the algorithm of Dong become unreliable when about 50% of the examined factors are significant. In such situation, it is advisable to perform an experimental design from which enough negligible effects can be estimated. The graphical interpretation method did not always succeed in indicating the correct number of significant effects.