Parallel Reinforcement Learning With Minimal Communication Overhead for IoT Environments

摘要

Many Internet of Things (IoT) applications require a distributed architecture for decision making either because of a lack of a centralized system, failure-prone connectivity to a centralized system or because the imposed latency to contact such a system is too high for real-time applications. Often, these IoT applications fall in the domain of reinforcement learning (RL), e.g., autonomous robot navigation in smart factories and traffic signal control in smart cities. However, RL-based applications require a long learning time. To overcome this limitation and scale with the number of agents, parallel RL (PRL) algorithms run multiple RL agents in parallel and on distributed environments. However, deploying PRL algorithms in such environments entails a communication overhead that increases the (actual) execution time. The state-of-the-art PRL algorithms are designed for reducing the learning time while assuming no (or limited) communication overhead. In this article, we present a novel partitioning algorithm that minimizes the communication overhead in PRL running on IoT environments. To the best of our knowledge, this is the first work that focuses on solving the communication overhead of distributing PRL algorithms without requiring any a priori knowledge about the structure of the problem. The proposed algorithm intelligently combines a dynamic state partitioning strategy, which exploits the agent's exploration capabilities to build partition knowledge while learning, with an efficient mapping of agents to partitions, which reduces the communication among agents. Performance evaluations show that the proposed algorithm can achieve almost no communication among PRL agents at the converged state.