Prototype filter design for QAM-based filter bank multicarrier system

In this paper, we provide conditions for a prototype filter design of filter bank multicarrier (FBMC) based on quadrature amplitude modulation (QAM). The conditions consist of a generalized Nyquist criterion (GNC) for nearly perfect reconstruction (NPR) and meeting the stopband condition for a small side-lobe. In a practical environment, a small side-lobe is a key aspect of achieving high spectral efficiency, which becomes an important factor in reducing the size of the guard band among channels. In addition, because the conventional GNC is derived under the assumption of an ideal channel, which can be easily broken over a practical multipath channel, we propose a relaxed NPR that considers the multipath delay using a 2L-oversampled discrete Fourier transform (DFT) in the frequency domain. Based on the relaxation of the GNC depending on a multipath channel, we formulate an optimization problem for a QAM-FBMC prototype filter design and propose a prototype filter with a small side-lobe and reliable BER performance. Simulation results show that the proposed prototype filter remarkably reduces the side-lobe compared to conventional QAM-FBMC based on two types of prototype filters and orthogonal frequency division multiplexing (OFDM) via the trade-off between GNC and side-lobe performance. The benefit of the side-lobe condition allows the proposed prototype filter, the Relaxed-NPR-F, to improve spectral efficiency by reducing the guard band in the frequency domain.