Approximate Soft Policy Iteration Based Reinforcement Learning for Differential Games with Two Pursuers versus One Evader

摘要

Pursuit-evasion is a well studied problem in the field of optimal control, differential game and game theory. In complex multi-pursuer environment with constraints, it is difficult to find optimal solution through traditional approaches. This paper establishes an obstacle-free bounded environment for two pursuers vs. one evader and develops an approximate soft policy iteration algorithm (ASPI) using a value neural network to provide cooperative policy for the pursuers. On this basis, the training process is composed of two phases with dense-sparse reward. The first phase uses dense reward to quickly obtain general ability of pursuing, and then the second phase uses sparse reward signal to shape the value function closer to the real distribution. The simulation results and comparative analysis with the classical deep Q-learning are displayed and prove the superior performance of this approach, in terms of the win rate of capture and the time cost of training.