Rational solitary wave and rogue wave solutions in coupled defocusing Hirota equation

•The M/W shape rational solitary wave solutions and rogue wave solutions of coupled Hirota equations are given.•The baseband modulational stability theory is established in the defocusing coupled Hirota model.•The M/W shape rational solitary wave can be explained by the baseband modulational stability theory.

We derive and study a general rational solution of a coupled defocusing Hirota equation which can be used to describe evolution of light in a two-mode fiber with defocusing Kerr effect and some certain high-order effects. We find some new excitation patterns in the model, such as M-shaped soliton, W-shaped soliton, anti-eye-shaped rogue wave and four-petaled flower rogue wave. The results are compared with the solutions obtained in other coupled systems like vector nonlinear Schrödinger equation, coupled focusing Hirota and Sasa–Satsuma equations. We explain the new characters by modulational instability properties. This further indicates that rational solution does not necessarily correspond to rogue wave excitation dynamics and the quantitative relation between nonlinear excitations and modulational instability should exist.