Rogue wave solutions of AB system

• The generalized Darboux transformation of the AB system is constructed.
• A unified formula of Nth-order rogue wave is obtained by the direct iterative rule.
• Rogue waves from first to second order are illustrated through some figures.

In this paper, the generalized Darboux transformation is established to the AB system, which mainly describes marginally unstable baroclinic wave packets in geophysical fluids and ultra-short pulses in nonlinear optics. A unified formula of Nth-order rogue wave solution for the AB system is found by the direct iterative rule. In particular, rogue wave solutions possessing several free parameters from first to second order are calculated. The dynamic properties of rogue waves in the AB system are shown through some figures.