Value-based Algorithms Optimization with Discounted Multiple-step Learning Method in Deep Reinforcement Learning

摘要

Value-based algorithms have been demonstrated on a range of deep reinforcement learning tasks. However, value-based algorithms suffer from challenges in the aspect of stability, overestimate and convergence, which limit severely the application of such algorithms in real-word environment. In n-step learning method, truncated N-step return was used as a part of multiple-step targets to make faster learning, and improve the defect of such algorithms, but it is still far away to practical application. In this paper, we proposed a straightforward optimal method — Discount Multiple-steps Learning Method (DMLM) to improve the performance of value-based algorithms by giving a discount factor to truncated N-step return which shows better results in our experiments. In this method, regard the discounted truncated N-step return rather than accumulated discount reward as the important part of target network when computing the TD-error as a loss function of evaluate network. In the experiment part, we perform experiments compare to value-based algorithms without this method, and prove this method can make more accurate predict of value function, thereby outperform other optimal methods in terms of stability, overestimate and convergence.