Learning task decomposition and exploration shaping for reinforcement learning agents

摘要

For situated reinforcement learning agents to succeed in complex real world environments they have to be able to efficiently acquire and reuse control knowledge in order to accomplish new tasks faster and to accelerate the learning of new policies. While hierarchical learning approaches which transfer previously acquired skills and representations to model and control new tasks have the potential to significantly improve learning times, they also pose the risk of “behavior proliferation” where the growing set of available actions makes it increasingly difficult to determine a strategy for a new task. To overcome this problem and to further improve knowledge reuse, the learning agent should thus also have the ability to predict the utility of an action or reusable skill in a new context and to analyze new tasks in order to decompose them into known subtasks. This paper presents a novel approach for learning task decomposition by learning to predict the utility of subgoals and subgoal types in the context of a new task, as well as for exploration shaping by predicting the likelihood with which each available action is useful in the given task context. This information, encoded as a set of utility functions, is then used to focus the exploration and learning process of the agent to increase performance both in terms of the time spent to reach the new task''s goal the first time and of the time required to learn an optimal policy. This ability is demonstrated here in the context of navigation and manipulation tasks in a feature enhanced grid world domain.