ACE (Actor–Critic–Explorer) paradigm for reinforcement learning in basal ganglia: Highlighting the role of subthalamic and pallidal nuclei

We present a comprehensive model of basal ganglia in which the three important reinforcement learning components—Actor, Critic and Explorer (ACE),—are represented and their anatomical substrates are identified. Particularly, we identify the subthalamic-nucleus and globus pallidus externa (STN–GPe) loop as the Explorer, and argue that complex activity of STN and GPe neurons, found in experimental studies, provides the stochastic drive necessary for exploration. Simulations involving a two-link arm model show task-dependent variations in complexity of STN–GPe activity when the ACE network is trained to perform simple reaching movements. Complexity and average levels of STN–GPe activity are observed to be higher before training than in post-training conditions. Further, in order to simulate Parkinsonian conditions, when dopamine levels in substantia nigra portion of the model are reduced, the arm displayed, as a primary change, small amplitude movements, which on persistent network training, amplified to large amplitude unregulated movements reminiscent of Parkinsonian tremor.