Direction-of-arrival estimation in a mixture of K-distributed and Gaussian noise

•We consider direction finding in a mixture of K and Gaussian noise.•Maximum likelihood estimation is considered with known or unknown textures.•A lower bound is derived.•The new estimators show improvement in high K to Gaussian noise ratio.

We address the problem of estimating the directions-of-arrival (DoAs) of multiple signals received in the presence of a combination of a strong compound-Gaussian external noise and weak internal white Gaussian noise. Since the exact distribution of the mixture is not known, we get an insight into optimum procedure via a related model where we consider the texture of the compound-Gaussian component as an unknown and deterministic quantity to be estimated together with DoAs or a basis of the signal subspace. Alternate maximization of the likelihood function is conducted and it is shown that it operates a separation between the snapshots with small/large texture values with respect to the additive noise power. The modified Cramér–Rao bound is derived and a prediction of the actual mean-square error is presented, based on separation between external/internal-noise dominated samples. Numerical simulations indicate that the suggested iterative DoA estimation technique comes close to the introduced bound and outperform a number of existing routines.