Evolution of social behavior in finite populations: A payoff transformation in general nn-player games and its implications

The evolution of social behavior has been the focus of many theoretical investigations, which typically have assumed infinite populations and specific payoff structures. This paper explores the evolution of social behavior in a finite population using a general nn-player game. First, we classify social behaviors in a group of nn individuals based on their effects on the actor’s and the social partner’s payoffs, showing that in general such classification is possible only for a given composition of strategies in the group. Second, we introduce a novel transformation of payoffs in the general nn-player game to formulate explicitly the effects of a social behavior on the actor’s and the social partners’ payoffs. Third, using the transformed payoffs, we derive the conditions for a social behavior to be favored by natural selection in a well-mixed population and in the presence of multilevel selection.