First Results with Instance-Based State Identification for Reinforcement Learning

摘要

When a reinforcement learning agent's next course of action depends oninformation that is hidden from the sensors because of problems such as occlusion, restricted range, bounded field of view and limited attention, we say the agent suffers from the hidden state problem. State identification techniques use history information to uncover hidden state. Previous approaches to encoding history include: finite state machines, recurrent neural networks, and genetic programming with indexed memory. A chief disadvantage of all these techniques is their long training time. This report presents instance-based state identification, a new approach to reinforcement learning with state identification that learns with much fewer training steps. Noting that learning with history and learning in continuous spaces both share the property that they begin without knowing the granularity of the state space, the approach applies instance-based (or memory-based) learning to history sequences-instead of recording instances in a continuous geometrical space, we record instances in action-perception-reward sequence space. The first implementation of this approach, called Nearest Sequence Memory, learns with an order of magnitude fewer steps than several previous approaches.