基于强化学习的 RoboCup Keepaway 高层抢球策略

摘要

In Robocop Keepaway training task,traditional hand-coded ball-stealing strategies are very subjective and can't adapt well to training situation changes,this leads to the takers taking longer time to complete the tasks and having lower stealing success rate.To solve this problem,we apply the reinforcement learning to high-level action decision-making for stealing takers in Keepaway task.By analysing the characteristic of stealing task,we reasonably design the state space,action space and reward value of the reinforcement learning model of stealing takers,and state a corresponding reinforcement learning algorithm for stealing takers.Experimental results show that after the rein-forced learning the stealing takers can make more objective decisions according to game's situation,the effect of the decisions made are re-markably better than the hand-coded strategies.For typical 4v3 and 5v4 scale Keepaway tasks,with the learned strategy to making decision, the stealing takers shorten 7.1% of the time at least for completing ball -stealing task,and the stealing success rate increases no less than 15.0% as well.%在 RoboCup Keepaway 任务训练中，传统手工抢球策略的主观性强，对训练情形变化的适应性差，导致抢球球员任务完成时间长、抢断成功率低。针对这一问题，将强化学习应用于 Keepaway 中抢球球员的高层动作决策。通过对抢球任务特点的分析，合理设计了抢球球员强化学习模型的状态空间、动作空间及回报值，并给出了抢球球员的强化学习算法。实验结果表明经强化学习后，抢球球员能够根据比赛情形做出更客观的决策，决策效果显著优于手工策略。对于4v3和5v4规模的典型 Keepaway 任务，抢球球员采用学习后的策略决策时，抢球任务完成时间至少缩短了7．1％，抢断成功率至少提升了15．0％。