Modulational instability in nonlinear nonlocal equations of regularized long wave type

• A modulational instability index for BBM type equations.
• A modulational instability index for regularized Boussinesq type equations.
• Modulational instability of supercritical small wave trains of the BBM equation.
• Modulational stability of small wave trains of the regularized Boussinesq equation.
• An instability diagram for fractional dispersion.

We study the stability and instability of periodic traveling waves in the vicinity of the origin in the spectral plane, for equations of Benjamin–Bona–Mahony (BBM) and regularized Boussinesq types permitting nonlocal dispersion. We extend recent results for equations of Korteweg–de Vries type and derive modulational instability indices as functions of the wave number of the underlying wave. We show that a sufficiently small, periodic traveling wave of the BBM equation is spectrally unstable to long wavelength perturbations if the wave number is greater than a critical value and a sufficiently small, periodic traveling wave of the regularized Boussinesq equation is stable to square integrable perturbations.