Localization of coherent signals without source number knowledge in unknown spatially correlated Gaussian noise

• Our solution can work properly in unknown spatially correlated Gaussian noise.
• Our solution can deal with correlated or even coherent signals.
• Our solution does not require any a priori information of the source number.

A new direction-of-arrival estimator for coherent signals in spatially correlated noise is devised in this paper. By constructing a set of fourth-order cumulant based Toeplitz matrices, the coherent signals can be decorrelated. Moreover, by utilizing the joint diagonalization structure of these Toeplitz matrices, a new cost function that does not require any a priori information of the source number is developed. Numerical examples are provided to demonstrate the effectiveness of the proposed approach.