Efficient source enumeration for accurate direction-of-arrival estimation in threshold region

• In the threshold region, nonlinear signal parameter estimates carry large mean square estimation error.
• For unequal-power and/or closely-spaced sources, the estimation errors for individual sources differ greatly.
• We propose the theoretical concept of effective source number which is useful for understanding what makes a good order selection rule in the threshold region.
• We also define the matched source number (MSN) for a parameter estimator in an effort to automatically remove inaccurate estimates while keeping as many accurate estimates as possible.
• We devise a signal number detection algorithm that attains the MSN.

Estimation of the number of signals impinging on an array of sensors, also known as source enumeration, is usually required prior to direction-of-arrival (DOA) estimation. In challenging scenarios such as the presence of closely-spaced sources and/or high level of noise, using the true source number for nonlinear parameter estimation leads to the threshold effect which is characterized by an abnormally large mean square error (MSE). In cases that sources have distinct powers and/or are closely spaced, the error distribution among parameter estimates of different sources is unbalanced. In other words, some estimates have small errors while others may be quite inaccurate with large errors. In practice, we will be only interested in the former and have no concern on the latter. To formulate this idea, the concept of effective source number (ESN) is proposed in the context of joint source enumeration and DOA estimation. The ESN refers to the actual number of sources that are visible at a given noise level by a parameter estimator. Given the numbers of sensors and snapshots, number of sources, source parameters and noise level, a Monte Carlo method is designed to determine the ESN, which is the maximum number of available accurate estimates. The ESN has a theoretical value in that it is useful for judging what makes a good source enumerator in the threshold region and can be employed as a performance benchmark of various source enumerators. Since the number of sources is often unknown, its estimate by a source enumerator is used for DOA estimation. In an effort to automatically remove inaccurate estimates while keeping as many accurate estimates as possible, we define the matched source number (MSN) as the one which in conjunction with a parameter estimator results in the smallest MSE of the parameter estimates. We also heuristically devise a detection scheme that attains the MSN for ESPRIT based on the combination of state-of-the-art source enumerators.