Multi-agent learning based routing for delay minimization in Cognitive Radio Networks

To overcome the problem of underutilizing licensed spectrum, Cognitive Radio Networks (CRNs) have emerged in which Secondary Users (SUs) are allowed to access opportunistically the licensed spectrum allocated exclusively to Primary Users (PUs). In the CRNs, routing process is essential to send data from source SUs to destination SUs. In the CRNs, a distributed routing solution should be considered for the CRNs, since the SUs use only local information to perform routing. Also, when there are multiple data flows generated by SUs, a multi-agent routing solution should be considered, since the routing performance of each SU is affected by routing decisions of other SUs. Recent real samples analysis has shown that the PUs spectrum usage pattern at the time domain exhibits the data traffic properties. So, we consider the data traffic for the PUs and the SUs, that have obtained incomplete information on PUs traffic through sensing, decide to exploit short lived spectrum holes between the PUs data packets for routing. In this paper, we introduce a distributed cooperative multi-agent routing problem in multi-hop CRNs, modeled using Decentralized Partially Observable Markov Decision Process (DEC-POMDP), in which the SUs want to minimize the end-to-end delay while keeping their interference to the PUs below a certain threshold. Then, a learning based scheme is developed for solving the problem. Simulation results demonstrate the convergence of the proposed learning based scheme and show that the performance of the proposed method is close to optimal method (that unlike the proposed method, assumes the SUs have complete information on PUs traffic and uses Q-learning for routing of SUs flows). Also, simulation results show that the proposed scheme maintains the end-to-end delay experienced by the packets in a low level and greatly outperforms related work at interference control.