The limiting spectral distribution of the product of the Wigner matrix and a nonnegative definite matrix

Let WnWn be n×nn×n Hermitian whose entries on and above the diagonal are independent complex random variables satisfying the Lindeberg type condition. Let TnTn be n×nn×n nonnegative definitive and be independent of WnWn. Assume that almost surely, as n→∞n→∞, the empirical distribution of the eigenvalues of TnTn converges weakly to a non-random probability distribution.Let An=n−1/2Tn1/2WnTn1/2. Then with the aid of the Stieltjes transforms, we show that almost surely, as n→∞n→∞, the empirical distribution of the eigenvalues of AnAn also converges weakly to a non-random probability distribution, a system of two equations determining the Stieltjes transform of the limiting distribution. Important analytic properties of this limiting spectral distribution are then derived by means of those equations. It is shown that the limiting spectral distribution is continuously differentiable everywhere on the real line except only at the origin and that a necessary and sufficient condition is available for determining its support. At the end, the density function of the limiting spectral distribution is calculated for two important cases of TnTn, when TnTn is a sample covariance matrix and when TnTn is the inverse of a sample covariance matrix.