Performance prediction of subspace-based adaptive detectors with signal mismatch

•We derive the distributions of the adaptive detectors with subspace signal mismatch.•We obtain the analytical expressions for the PDs and PFAs of the detectors.•This extends the existing theory for rank-one signal detection in the mismatch case.

For the problem of subspace signal detection, three adaptive detectors have been proposed in the past, namely, the subspace-based generalized likelihood ratio test, subspace-based adaptive matched filter, and adaptive subspace detector. In this paper we analyze their performance in the mismatch case, where the actual signal does not exactly lie in the nominal signal subspace. We derive their statistical distributions, and then obtain analytical expressions for the probabilities of detection and false alarm. It is shown that the signal mismatch has a significant effect on their detection performance through a quantity, which can be taken as a measure of the “distance” between the actual signal and the nominal signal subspace. These results extend the existing theory for rank-one signal detection in the mismatch case.