Reinforcement learning in population games

We study reinforcement learning in a population game. Agents in a population game revise mixed strategies using the Cross rule of reinforcement learning. The population state-the probability distribution over the set of mixed strategies-evolves according to the replicator continuity equation which, in its simplest form, is a partial differential equation. The replicator dynamic is a special case in which the initial population state is homogeneous, i.e. when all agents use the same mixed strategy. We apply the continuity dynamic to various classes of symmetric games. Using 3×3 coordination games, we show that equilibrium selection depends on the variance of the initial strategy distribution, or initial population heterogeneity. We give an example of a 2×2 game in which heterogeneity persists even as the mean population state converges to a mixed equilibrium. Finally, we apply the dynamic to negative definite and doubly symmetric games.

► Agents in a population game revise mixed strategies using reinforcement learning. ► The strategy distribution evolves according to the replicator continuity equation. ► The replicator dynamic is a special case when all agents use same mixed strategy. ► Equilibrium selection depends on the variance of the initial strategy distribution. ► Even if average strategy approaches Nash equilibrium, variance may persist.