Localization of coherent signals without source number knowledge in unknown spatially correlated Gaussian noise

•Our solution can work properly in unknown spatially correlated Gaussian noise.•Our solution can deal with correlated or even coherent signals.•Our solution does not require any a priori information of the source number.

A new direction-of-arrival estimator for coherent signals in spatially correlated noise is devised in this paper. By constructing a set of fourth-order cumulant based Toeplitz matrices, the coherent signals can be decorrelated. Moreover, by utilizing the joint diagonalization structure of these Toeplitz matrices, a new cost function that does not require any a priori information of the source number is developed. Numerical examples are provided to demonstrate the effectiveness of the proposed approach.