An adaptive penalized maximum likelihood algorithm

The LMS algorithm is one of the most popular learning algorithms for identifying an unknown system. Many variants of the algorithm have been developed based on different problem formulations and principles. In this paper, we use the penalized maximum likelihood (PML) as a principled and unified approach for developing LMS-type algorithms. We study a general solution to the problem and develop algorithms to address the problems of robustness to impulsive noise and exploiting the sparseness of the system. We perform a statistical analysis of a special case of the proposed algorithm and propose a data-driven method to update the penalty parameter. We also reveal an invariant property of the algorithm. Connections with algorithms based on stochastic gradient descent are also studied. We demonstrate the competitive performance of the proposed algorithms by numerical examples and comparison with recently published algorithms.