Predator–prey systems with a general non-monotonic functional response

•We derive a game theoretic model of prey group defense behavior.•The model predicts that a predator’s functional response can be non-monotonic.•We analyze the predator–prey dynamics with a general non-monotonic functional response.•We present the Reduced Morse–Smale portrait and conditions for a limit cycle.

In this paper we study a predator–prey model with a general non-monotonic functional response. We first demonstrate that the non-monotonicity can be explained by a tradeoff argument using game theory. We then study the Reduced Morse–Smale portrait of the general system, that is all the possible stable phase portraits without limit cycles. We show that under suitable conditions the system admits at least one limit cycle.