Utilizing kinematics and selective sweeping in reinforcement learning-based routing algorithms for underwater networks

Effective utilization of mobile ad hoc underwater distributed networks is challenging due to high system costs and the harsh environment characterized by low bandwidth, large latency, high energy consumption, and node mobility. This work addresses the routing issue, which is critical in successfully establishing and utilizing an underwater network. In particular, it focuses on reinforcement learning (RL)-based routing algorithms, which possess the ability to explore the network environment and adapt routing decisions to the constantly changing topology of the network due to node mobility and energy usage. This paper presents a routing algorithm based on Q-learning, one of the RL approaches, with additional Kinematic and Sweeping features, therefore referred to as QKS. These two additional features are introduced to address the potential slow convergence associated with pure RL algorithms. The results of a detailed packet-level simulation have been obtained using the NS-2 open-source network simulator with underwater modeling additions. The energy efficiency, convergence, and delivery performance of QKS are compared with two other routing protocols for underwater networks, a basic flooding approach (ICRP (Liang, 2007)) and a basic Q-learning implementation (QELAR (Hu, 2010)), using simulations of networks with both fixed and mobile nodes.