A biologically motivated synthesis of accumulator and reinforcement-learning models for describing adaptive decision-making

摘要

Cognitive process models, such as reinforcement learning (RL) and accumulator models of decision-making, have proven to be highly insightful tools for studying adaptive behaviors as well as their underlying neural substrates. Currently, however, two major barriers exist preventing these models from being applied in more complex settings: 1) the assumptions of most accumulator models break down for decisions involving more than two alternatives; 2) RL and accumulator models currently exist as separate frameworks, with no clear mapping between trial-to-trial learning and the dynamics of the decision process. Recently I showed how a modified accumulator model, premised off of the architecture of cortico-basal ganglia pathways, both predicts human decisions in uncertain situations and evoked activity in cortical and subcortical control circuits. Here I present a synthesis of RL and accumulator models that is motivated by recent evidence that the basal ganglia acts as a site for integrating trial-wise feedback from midbrain dopaminergic neurons with accumulating evidence from sensory and associative cortices. I show how this hybrid model can explain both adaptive go/no-go decisions and multi-alternative decisions in a computationally efficient manner. More importantly, by parameterizing the model to conform to various underlying assumptions about the architecture and physiology of basal ganglia pathways, model predictions can be rigorously tested against observed patterns in behavior as well as neural recordings. The result is a biologically-constrained and behaviorally tractable description of trial-to-trial learning effects on decision-making among multiple alternatives.