Iterative nonlinear ISI cancellation in optical tilted filter-based Nyquist 4-PAM system

• Improve the system’s frequency response.
• Extend the transmission distance.
• Mitigate power fading and nonlinear inter-symbol interference.

The conventional double sideband (DSB) modulation and direct detection scheme suffers from severer power fading, linear and nonlinear inter-symbol interference (ISI) caused by fiber dispersion and square-law direct detection. The system’s frequency response deteriorates at high frequencies owing to the limited device bandwidth. Moreover, the linear and nonlinear ISI is enhanced induced by the bandwidth limited effect. In this paper, an optical tilted filter is used to mitigate the effect of power fading, and improve the high frequency response of bandwidth limited device in Nyquist 4-ary pulse amplitude modulation (4-PAM) system. Furtherly, iterative technique is introduced to mitigate the nonlinear ISI caused by the combined effects of electrical Nyquist filter, limited device bandwidth, optical tilted filter, dispersion, and square-law photo-detection. Thus, the system’s frequency response is greatly improved and the delivery distance can be extended.