Modified likelihood ratio test in finite mixture models with a structural parameter

The finite mixture model is an example of a non-regular parametric family, and most classical asymptotic results cannot be directly applied. In particular, the asymptotic properties of likelihood ratio statistics for testing for the number of subpopulations are complicated and difficult to establish. One approach that has been found to simplify the asymptotic results while preserving the power of the test is to modify the likelihood function by incorporating a penalty term to avoid boundary problems. The asymptotic properties and the use of likelihood ratio results are even more difficult when an unknown structural parameter is involved in the model. In this paper, we study an application of the modified likelihood approach to finite normal mixture models with a common and unknown variance in the mixing components and consider a test of the hypothesis of a homogeneous model versus a mixture on two or more components. We show that the χ22 distribution is a stochastic lower bound to the limiting distribution of the likelihood ratio statistic. This same distribution is also shown to be a stochastic upper bound to the limiting distribution of the modified likelihood ratio statistic. A small simulation study suggests that both bounds are relatively tight and practically useful. An example from genetics is used to illustrate the technique.