Transition, coexistence, and interaction of vector localized waves arising from higher-order effects

•Vector rogue wave properties induced by higher-order effects are studied.•A transition between vector rogue waves and solitons is obtained.•The link between the transition and modulation instability (MI) is demonstrated.•The coexistence of vector solitons and breathers coincides with the MI features.•An annihilation phenomenon for the vector two w-shaped solitons is presented.

We study vector localized waves on continuous wave background with higher-order effects in a two-mode optical fiber. The striking properties of transition, coexistence, and interaction of these localized waves arising from higher-order effects are revealed in combination with corresponding modulation instability (MI) characteristics. It shows that these vector localized wave properties have no analogues in the case without higher-order effects. Specifically, compared to the scalar case, an intriguing transition between bright–dark rogue waves and w-shaped–anti-w-shaped solitons, which occurs as a result of the attenuation of MI growth rate to vanishing in the zero-frequency perturbation region, is exhibited with the relative background frequency. In particular, our results show that the w-shaped–anti-w-shaped solitons can coexist with breathers, coinciding with the MI analysis where the coexistence condition is a mixture of a modulation stability and MI region. It is interesting that their interaction is inelastic and describes a fusion process. In addition, we demonstrate an annihilation phenomenon for the interaction of two w-shaped solitons which is identified essentially as an inelastic collision in this system.