Size and power of tests for a zero random effect variance or polynomial regression in additive and linear mixed models

Several tests for a zero random effect variance in linear mixed models are compared. This testing problem is non-regular because the tested parameter is on the boundary of the parameter space. Size and power of the different tests are investigated in an extensive simulation study that covers a variety of important settings. These include testing for polynomial regression versus a general smooth alternative using penalized splines. Among the test procedures considered, three are based on the restricted likelihood ratio test statistic (RLRT  ), while six are different extensions of the linear model FF-test to the linear mixed model. Four of the tests with unknown null distributions are based on a parametric bootstrap, the other tests rely on approximate or asymptotic distributions. The parametric bootstrap-based tests all have a similar performance. Tests based on approximate FF-distributions are usually the least powerful among the tests under consideration. The chi-square mixture approximation for the RLRT is confirmed to be conservative, with corresponding loss in power. A recently developed approximation to the distribution of the RLRT is identified as a rapid, powerful and reliable alternative to computationally intensive parametric bootstrap procedures. This novel method extends the exact distribution available for models with one random effect to models with several random effects.