Building a Library of Policies through Policy Reuse

摘要

Policy Reuse (PR) provides Reinforcement Learning algorithms with a mechanism to bias an exploration process by reusing a set of past policies. Policy Reuse offers the challenge of balancing the exploitation of the ongoing learned policy, the exploration of new random actions, and the exploitation of past policies. Efficient application of Policy Reuse requires a mechanism to build, for each domain, a library of policies that is useful and accurate enough to efficiently solve any task in such a domain. In this work, the authors propose a mechanism to create a library of policies based on a similarity metric among policies. If the new policy is similar to any of the past ones, it is not added to the library. Otherwise, it is stored together with the other policies so it can be reused in the future. Thus, the Policy Library stores the 'basis' or 'eigen-policies' of each domain (i.e., the core past policies that are effectively reusable). Empirical results demonstrate that the Policy Library can be efficiently created and that the stored 'eigen- policies' can be understood as a representation of the structure of the domain.