Fourth-order cumulants-based sparse representation approach for DOA estimation in MIMO radar with unknown mutual coupling

• The DOA estimation problem for monostatic MIMO radar with unknown mutual coupling is considered.
• Fourth-order cumulant matrix with advantageous formation is constructed to suppress colored Gaussian noises.
• A reweighted sparse representation framework of FOC matrix is proposed for the accurate DOA estimation.
• The proposed method provides better performance than ESPRIT-Like, l1-SVDl1-SVD and FOC-MUSIC algorithms in both white and colored Gaussian noise conditions.

In this paper, a sparse representation approach based on fourth-order cumulants (FOC) is proposed for direction of arrival (DOA) estimation in monostatic multiple-input multiple-output (MIMO) radar with unknown mutual coupling. For applying the sparse representation theory successfully, exploiting the special banded symmetric Toeplitz structure of mutual coupling matrices (MCM) in both transmit array and receive array, the unknown MCM in received data can be turned into a diagonal one to eliminate the mutual coupling. Then based on the new received data, a reduced dimensional transformation matrix is formulated, and the proposed method further constructs a FOC matrix with special formation, which reduce the computational complexity of sparse signal reconstruction. Finally a reweighted l1-norm constraint minimization sparse representation framework is designed, and the DOAs can be obtained by finding the non-zero rows in the recovered matrix. Owing to utilizing the fourth-order cumulants and reweighted sparse representation framework, compared with ESPRIT-Like, FOC-MUSIC and l1-SVD algorithms, the proposed method performs well in both white and colored Gaussian noise conditions, meanwhile it has higher angular resolution and better angle estimation performance. Simulation results verify the effectiveness and advantages of the proposed method.