Improved Bayesian information criterion for mixture model selection

•A new criterion for mixture model selection is proposed.•Mathematical derivation of the criterion is justified.•The proposed criterion works as good as the state-of-the-art criteria for large sample size.•The proposed criterion outperforms the state-of-the-art criteria for small sample size.•The proposed criterion performs well for real datasets.

In this paper, we propose a mixture model selection criterion obtained from the Laplace approximation of marginal likelihood. Our approximation to the marginal likelihood is more accurate than Bayesian information criterion (BIC), especially for small sample size. We show experimentally that our criterion works as good as other well-known criteria like BIC and minimum message length (MML) for large sample size and significantly outperforms them when fewer data points are available.