Fish–hook shaped global bifurcation branch of a spatially heterogeneous cooperative system with cross-diffusion

In this paper, we will consider the following strongly coupled cooperative system in a spatially heterogeneous environment with Neumann boundary condition{Δu+u(λ−u+b(x)v)=0,x∈Ω,Δ[(1+kρ(x)u)v]+v(μ−v+d(x)u)=0,x∈Ω,∂νu=∂νv=0,x∈∂Ω, where Ω   is a bounded domain in RNRN (N⩾1N⩾1) with smooth boundary ∂Ω; k is a positive constant, λ and μ   are real constants which may be non-positive; b(x)⩾≢0b(x)⩾≢0 and d(x)⩾≢0d(x)⩾≢0 are continuous functions in Ω¯; ρ(x)ρ(x) is a smooth positive function in Ω¯ with ∂νρ(x)|∂Ω=0∂νρ(x)|∂Ω=0; ν is the outward unit normal vector on ∂Ω   and ∂ν=∂/∂ν∂ν=∂/∂ν. For the case μ>0μ>0, we show that if |μ||μ| is small and k   is large, a spatial segregation of ρ(x)ρ(x) and b(x)b(x) can cause the positive solution curve to form an unbounded fish–hook (⊂) shaped curve with parameter λ  . For the case μ<0μ<0, if |μ||μ| is small and k is large, and the cooperative effect is strong for species u and very weak for species v, then the positive solution set also forms an unbounded fish–hook shaped continuum. These results are quite different from those of predator–prey systems and the cooperative system under Dirichlet boundary condition, both of which can form a bounded continuum. Our results deduce that the spatial heterogeneity of environments can produce multiple coexistence states. Our method of analysis is based on the bifurcation theory and the Lyapunov–Schmidt procedure.