Train Small, Deploy Big: Do Relative World Views Permit Swarm-Safety During Policy Transplantation for Multi-Agent Reinforcement Learning Problems?

摘要

In order to 'train small, deploy big', agent control policies must be transplanted from one trained agent into a larger set of agents for deployment. Given that compute resources and training time generally scale with the number of agents, this approach to generating swarm control policies may be favourable for larger swarms. However, in order for this process to be successful, the agent control policy must be indistinct to the agent on which it is trained so that it can perform as required in its new host agent. Through extensive simulation of a cooperative multi-agent navigation task, it is shown that this indistinctness of agent policies, and therefore the success of the associated learned solution of the transplanted swarm, is dependent upon the way in which an agent views the world: absolute or relative. As a corollary to, and in contrary to naive intuition of, this result, we show that homogeneous agent capability is not enough to guarantee policy indistinctness. The article also discusses what general conditions may be required in order to enforce policy indistinctness.