Robust estimation in the normal mixture model

This paper focuses on the inference of the normal mixture model with unequal variances. A feature of the model is flexibility of density shape, but its flexibility causes the unboundedness of the likelihood function and excessive sensitivity of the maximum likelihood estimator to outliers. A modified likelihood approach suggested in Basu et al. [1998, Biometrika 85, 549–559] can overcome these drawbacks. It is shown that the modified likelihood function is bounded above under a mild condition on mixing proportions and the resultant estimator is robust to outliers. A relationship between robustness and efficiency is investigated and an adaptive method for selecting the tuning parameter of the modified likelihood is suggested, based on the robust model selection criterion and the cross-validation. An EM-like algorithm is also constructed. Numerical studies are presented to evaluate the performance. The robust method is applied to single nuleotide polymorphism typing for the purpose of outlier detection and clustering.