Energy-Efficient Resource Allocation in Cognitive Radio Networks Under Cooperative Multi-Agent Model-Free Reinforcement Learning Schemes

摘要

The most prominent challenge to the wireless community is to meet the demand for radio resources. Cognitive Radio (CR) is envisioned as a potential solution that utilizes its cognition ability intended to enhance the proper utilization of available radio resources and improves energy efficiency. However, due to the co-existence of Primary Base Stations (PU-BSs) and Cognitive Base Stations (CR-BSs) in CR networks, the problem of aggregated interference occurs which poses a critical challenge for resource allocation in CR networks. Moreover, in practical scenarios, it is difficult to form the correct network model due to complex network dynamics beforehand. Therefore, this work presents Multi-Agent Model-Free Reinforcement Learning schemes namely Q-Learning (Q-L) and State-Action-Reward- (next) State- (next) Action (SARSA) for resource allocation which mitigates interference and eliminate the need of network model. The proposed schemes are implemented in a decentralized cooperative manner with CRs act as multi-agent, forms a stochastic dynamic team to obtain optimal energy-efficient resource allocation strategy. Numerical results reveal that: 1) proposed cooperative scheme 1 (Cooperative Q-L scheme) expedites the convergence; 2) proposed cooperative scheme 2 (Cooperative SARSA scheme) achieves significant improvement in network capacity. Both the proposed cooperative schemes demonstrate its effectiveness by providing significant improvement in energy efficiency and maintain users' QoS.