Multi-agent learning algorithms for content placement in cache-enabled small cell networks: 4G and 5G use cases

摘要

Caching popular files at the small base stations (SBSs) has proved to be an effective strategy to reduce the content delivery delay in cellular networks and to alleviate the backhaul congestion. In the optimization of the placement of contents into SBS caches (the so-called content placement problem), several key parameters play an important role, such as content popularity, the mobile users' (MUs') channel state information (CSI), as well as the capacity of the backhaul links. These parameters are random in general, and their instantaneous values over time give rise to a stochastic process. In this paper, we propose a mathematical formulation for the distributed optimization of content placement with the objective of minimizing the average content delivery latency. Our formulation is applicable to both conventional 4G small cell networks (SCNs) as well as 5G-compatiable mmWave integrated access and backhaul (IAB) cellular communications. In particular, the placement problem is modeled as a potential game among SBSs in which the objective of each SBS is to minimize the average delay of the MUs within its coverage range. In order to compute the Nash equilibrium (NE) of the game, we adopt the learning-theoretic approach that only relies on incomplete information (or implicit feedback) of the system's underlying stochastic processes; i.e., the content placement is optimized in run-time by gaining experience and through the immediate noisy feedbacks of the actions actually taken in the operating environment. We propose an algorithm based on multi-agent reinforcement learning (MARL) techniques for potential games. It operates in the independent action space and can learn the optimal strategy profile of the SBSs in larger-scale scenarios, even when the actions of its peers are not observable by each SBS. Simulation experiments are conducted to investigate the convergence of the learning algorithm as well as to compare against some schemes using prior knowledge.