Modeling and estimating passive dynamics distributions in linearly solvable Markov decision processes

摘要

Todorovにより，強化学習が簡単化されるマルコフ決定過程のクラスが提案されている．すなわち，特定の条件下では，最適行動選択のためのベルマン方程式が線形になり，最適な状態遷移確率が解析的に得られる．しかしながら，実問題に対して線形可解マルコフ決定過程の枠組みを適用するには，システムの受動ダイナミクスを知る必要がある．本研究では，強化学習においてシステムのダイナミクスを推定する方法を提案する．%Todorov has recently introduced a class of linearly-solvable Markov decision processes (LSMDPs) which greatly simplifies reinforcement learning. Under some specific conditions, the problem of choosing optimal actions becomes linear, and the optimal transition probabilities can be obtained analytically. In order to apply the LSMDPs framework to realistic problems, it is necessary to know the passive dynamics distribution, which is crucial in the theory. The purpose of the present work is to propose a method to estimate the passive dynamics distribution in reinforcement learning problems.