BSE-MAML: Model Agnostic Meta-Reinforcement Learning via Bayesian Structured Exploration

摘要

Deep reinforcement learning (RL) is playing an increasingly important role in web services such as news recommendation, vulnerability detection, and personalized services. Exploration is a key component of RL, which determines whether these RL-based applications could find effective solutions eventually. In this paper, we propose a novel gradient–based fast adaptation approach for model agnostic meta-reinforcement learning via Bayesian structure exploration (BSE-MAML). BSE-MAML could effectively learn exploration strategies from prior experience by updating policy with embedding latent space via a Bayesian mechanism. Coherent stochasticity injected by latent space are more efficient than random noise, and can produce exploration strategies to perform well in novel environment. We have conducted extensive experiments to evaluate BSE-MAML. Experimental results show that BSE-MAML achieves better performance in exploration in realistic environments with sparse rewards, compared to state-of-the-art meta-RL algorithms, RL methods without learning exploration strategies, and task-agnostic exploration approaches.