Transition, coexistence, and interaction of vector localized waves arising from higher-order effects

• Vector rogue wave properties induced by higher-order effects are studied.
• A transition between vector rogue waves and solitons is obtained.
• The link between the transition and modulation instability (MI) is demonstrated.
• The coexistence of vector solitons and breathers coincides with the MI features.
• An annihilation phenomenon for the vector two w-shaped solitons is presented.

We study vector localized waves on continuous wave background with higher-order effects in a two-mode optical fiber. The striking properties of transition, coexistence, and interaction of these localized waves arising from higher-order effects are revealed in combination with corresponding modulation instability (MI) characteristics. It shows that these vector localized wave properties have no analogues in the case without higher-order effects. Specifically, compared to the scalar case, an intriguing transition between bright–dark rogue waves and w-shaped–anti-w-shaped solitons, which occurs as a result of the attenuation of MI growth rate to vanishing in the zero-frequency perturbation region, is exhibited with the relative background frequency. In particular, our results show that the w-shaped–anti-w-shaped solitons can coexist with breathers, coinciding with the MI analysis where the coexistence condition is a mixture of a modulation stability and MI region. It is interesting that their interaction is inelastic and describes a fusion process. In addition, we demonstrate an annihilation phenomenon for the interaction of two w-shaped solitons which is identified essentially as an inelastic collision in this system.