Solving the distal reward problem through linkage of STDP and dopamine signaling

摘要

In Pavlovian and instrumental conditioning, rewards typically come seconds after reward-triggering actions, creating an explanatory conundrum known as the distal reward problem or the credit assignment problem. How does the brain know what firing patterns of what neurons are responsible for the reward if (1) the firing patterns are no longer there when the reward arrives and (2) most neurons and synapses are active during the waiting period to the reward? A model network and computer simulation of cortical spiking neurons with spike-timing-dependent plasticity (STDP) modulated by dopamine (DA) is disclosed to answer this question. STDP is triggered by nearly-coincident firing patterns of a presynaptic neuron and a postsynaptic neuron on a millisecond time scale, with slow kinetics of subsequent synaptic plasticity being sensitive to changes in the extracellular dopamine DA concentration during the critical period of a few seconds after the nearly-coincident firing patterns. Random neuronal firings during the waiting period leading to the reward do not affect STDP, and hence make the neural network insensitive to this ongoing random firing activity. The importance of precise firing patterns in brain dynamics and the use of a global diffusive reinforcement signal in the form of extracellular dopamine DA can selectively influence the right synapses at the right time.