不確実な報酬予測におけるドーパミン活動の計算論的モデル

摘要

動物にとって，不確かな環境で生存するためには，未来に与えられる報酬をできるだけ正確に予測することが重要である.霊長類の中脳にあるドーパミン作動性（DA）細胞は，報酬予測に関与し，学習や連動制御にも関わることから多くの生理学的，計算論的研究がなされている．計算論的には，DA細胞の活動は，強化学習におけるTD誤差を表現するという仮説が提案されている．しかし近年，確率的報酬課題におけるDA細胞の活動頻度が，実際の報酬を得る時刻に向けてなだらかに増加することが報告されており，これは単純なTDモデルでは一見説明できないことから，議論を呼んでいる.本研究では，確率的報酬課題に内在する不確かさに着目し，DA細胞の予測的な活動度上昇を説明できる計算論的モデルを提案する.計算機実験により，確率的定式化によって定義された期待誤差が，報酬待ち時刻のDA細胞の活動をより良く説明できることと，さらに単試行の結果をも再現できることを示す．%In order for animals to behave effectively in their surrounding uncertain environments, it is essentially important to predict future outcomes as accurately as possible. Dopaminergic (DA) neurons in the primate mid-brain have been known to be involved in the brain's reward system and in many brain functions including learning and motor control, and therefore many physiological and computational studies have investigated the role of DA neurons. Prom a computational perspective, phasic activities of DA neurons have been considered as representing temporal difference (TD) errors, a learning signal in reinforcement learning. Recently, however, several studies have reported that, in stochastic reward tasks, the DA activities gradually increase before receiving actual rewards, which cannot be well explained by the simple TD model. In this study, we propose an alternative model based on a probabilistic formulation of the stochastic reward task to explain the predictive increase of DA activities. In simulation experiments, expectation errors denned by the probabilistic modeling, well described the gradually increasing DA activities during a wait period even in a single trial.