A novel square-root Nyquist filter design with prescribed ISI energy

••A method to design minimum phase square Nyquist filter with prescribed ISI energy is given.••Lagrange multipliers method and generalized singular value decomposition are used to develop this method.••An iterative weighted least squares method is used to make the stop band of the filter equiripple.••The proposed method only requires simple linear algebraic numerical computations.

In this paper, a novel method is introduced to design Nyquist filters with prescribed inter symbol interference (ISI) energy. The computation is based on the iterative weighted least-squares optimization with a quadratic constraint. The weighted stop band and the tail energies of the Nyquist filter are minimized iteratively while the quadratic constraint prescribing the ISI energy is enforced. The Lagrange multiplier method and generalized singular value decomposition are utilized to solve the problem. By changing the weighting coefficient, one can make a trade-off between the stop band energy and the tail energy (robustness to timing jitter) while the ISI energy is predetermined. The designed Nyquist filter can be factorized to a minimum phase and a maximum phase FIR pulse shaping square-root Nyquist filters for the transmitter and receiver with a simple modification.