Outlier detection and robust mixture modeling using nonconvex penalized likelihood

•We propose a robust mixture modeling approach using a mean-shift formulation coupled with nonconvex sparsity-inducing penalization, to conduct simultaneous outlier detection and robust parameter estimation.•We propose a general scale-free and case-specific mean-shift formulation to solve the general case of unequal component variances for mixture models.•An efficient iterative thresholding-embedded EM algorithm is developed to maximize the penalized log-likelihood.•The efficacy of the proposed approach is demonstrated via simulation studies and a real application on Acidity data analysis.

Finite mixture models are widely used in a variety of statistical applications. However, the classical normal mixture model with maximum likelihood estimation is prone to the presence of only a few severe outliers. We propose a robust mixture modeling approach using a mean-shift formulation coupled with nonconvex sparsity-inducing penalization, to conduct simultaneous outlier detection and robust parameter estimation. An efficient iterative thresholding-embedded EM algorithm is developed to maximize the penalized log-likelihood. The efficacy of our proposed approach is demonstrated via simulation studies and a real application on Acidity data analysis.