Effects of intermode nonlinearity and intramode nonlinearity on modulation instability in randomly birefringent two-mode optical fibers

We analyze in detail the effects of the intermode nonlinearity (IEMN) and intramode nonlinearity (IRMN) on modulation instability (MI) in randomly birefringent two-mode optical fibers (RB-TMFs). In the anomalous dispersion regime, the MI gain enhances significantly as the IEMN and IRMN coefficients increases. In the normal dispersion regime, MI can be generated without the differential mode group delay (DMGD) effect, as long as the IEMN coefficient between two distinct modes is above a critical value, or the IRMN coefficient inside a mode is below a critical value. This critical IEMN (IRMN) coefficient depends strongly on the given IRMN (IEMN) coefficient and DMGD for a given nonlinear RB-TMF structure, and is independent on the input total power, the power ratio distribution and the group velocity dispersion (GVD) ratio between the two modes. On the other hand, in contrast to the MI band arising from the pure effect of DMGD in the normal dispersion regime, where MI vanishes after a critical total power, the generated MI band under the combined effects of IEMN and IRMN without DMGD exists for any total power and enhances with the total power. The MI analysis is verified numerically by launching perturbed continuous waves (CWs) with wave propagation method.