On the characterization of breather and rogue wave solutions and modulation instability of a coupled generalized nonlinear Schrödinger equations

•We investigate a CGNLS system which includes four wave mixing (FWM) effects.•We construct rogue wave (RW) solutions of CGNLS system.•We show that RW occurs only when FWM parameter is real.•We observe modulation instability (MI) happens only for real value of FWM parameter.•Our result gives an evidence for the connection between occurrence of RW and MI.

We construct Darboux transformation of a coupled generalized nonlinear Schrödinger (CGNLS) equations and obtain exact analytic expressions of breather and rogue wave solutions. We also formulate the conditions for isolating these solutions. We show that the rogue wave solution can be found only when the four wave mixing parameter becomes real. We also investigate the modulation instability of the steady state solution of CGNLS system and demonstrate that it can occur only when the four wave mixing parameter becomes real. Our results give an evidence for the connection between the occurrence of rogue wave solution and the modulation instability.