On the number of limit cycles in small perturbations of a class of hyper-elliptic Hamiltonian systems with one nilpotent saddle

In this paper, we make a complete study on small perturbations of Hamiltonian vector field with a hyper-elliptic Hamiltonian of degree five, which is a Liénard system of the form x′=y, y′=Q1(x)+εyQ2(x) with Q1 and Q2 polynomials of degree respectively 4 and 3. It is shown that this system can undergo degenerated Hopf bifurcation and Poincaré bifurcation, which emerges at most three limit cycles in the plane for sufficiently small positive ε. And the limit cycles can encompass only an equilibrium inside, i.e. the configuration (3,0) of limit cycles can appear for some values of parameters, where (3,0) stands for three limit cycles surrounding an equilibrium and no limit cycles surrounding two equilibria.