Behavioral refuges and predator–prey coexistence

The effects of a behavioral refuge caused either by the predator optimal foraging or prey adaptive antipredator behavior on the Gause predator–prey model are studied. It is shown that both of these mechanisms promote predator–prey coexistence either at an equilibrium, or along a limit cycle. Adaptive prey refuge use leads to hysteresis in prey antipredator behavior which allows predator–prey coexistence along a limit cycle. Similarly, optimal predator foraging leads to sigmoidal functional responses with a potential to stabilize predator–prey population dynamics at an equilibrium, or along a limit cycle.

► Optimal predator foraging or optimal predator avoidance by prey creates a behavioral refuge for prey in predator–prey models.
► Such a behavioral refuge promotes predator–prey coexistence in the Gause predator–prey model.
► Predator avoidance by prey leads to a game that has two evolutionarily stable strategies at current population densities.
► The existence of these ESS leads to a hysteresis in prey behavior.