On the method of approximate Fisher scoring for finite mixtures of multinomials

Finite mixture distributions arise naturally in many applications including clustering and inference in heterogeneous populations. Such models usually do not yield closed formulas for maximum likelihood estimates, hence numerical methods such as the well-known Fisher scoring or Expectation–Maximization (EM) algorithms are used in practice. This work considers an approximate Fisher scoring algorithm (AFSA) which has previously been used to fit the binomial finite mixture and a special multinomial finite mixture designed to handle extra variation. AFSA iterations are based on a certain matrix which approximates the Fisher information matrix. First focusing on the general finite mixture of multinomials, we show that the AFSA approach is closely related to Expectation–Maximization, and can similarly be generalized to other finite mixtures and other missing data problems. Like EM, AFSA is more robust to the choice of initial value than Fisher scoring. A hybrid of AFSA and classical Fisher scoring iterations provides the best of both computational efficiency and stable convergence properties.