Constrained Deep Reinforcement Learning for Energy Sustainable Multi-UAV based Random Access IoT Networks with NOMA

by   Sami Khairy, et al.

In this paper, we apply the Non-Orthogonal Multiple Access (NOMA) technique to improve the massive channel access of a wireless IoT network where solar-powered Unmanned Aerial Vehicles (UAVs) relay data from IoT devices to remote servers. Specifically, IoT devices contend for accessing the shared wireless channel using an adaptive p-persistent slotted Aloha protocol; and the solar-powered UAVs adopt Successive Interference Cancellation (SIC) to decode multiple received data from IoT devices to improve access efficiency. To enable an energy-sustainable capacity-optimal network, we study the joint problem of dynamic multi-UAV altitude control and multi-cell wireless channel access management of IoT devices as a stochastic control problem with multiple energy constraints. To learn an optimal control policy, we first formulate this problem as a Constrained Markov Decision Process (CMDP), and propose an online model-free Constrained Deep Reinforcement Learning (CDRL) algorithm based on Lagrangian primal-dual policy optimization to solve the CMDP. Extensive simulations demonstrate that our proposed algorithm learns a cooperative policy among UAVs in which the altitude of UAVs and channel access probability of IoT devices are dynamically and jointly controlled to attain the maximal long-term network capacity while maintaining energy sustainability of UAVs. The proposed algorithm outperforms Deep RL based solutions with reward shaping to account for energy costs, and achieves a temporal average system capacity which is 82.4% higher than that of a feasible DRL based solution, and only 6.47% lower compared to that of the energy-constraint-free system.


page 1

page 7

page 11

page 14


Learning-Based Distributed Random Access for Multi-Cell IoT Networks with NOMA

Non-orthogonal multiple access (NOMA) is a key technology to enable mass...

Multi-Agent Graph Reinforcement Learning based On-Demand Wireless Energy Transfer in Multi-UAV-aided IoT Network

This paper proposes a new on-demand wireless energy transfer (WET) schem...

Catch Me If You Can: Deep Meta-RL for Search-and-Rescue using LoRa UAV Networks

Long range (LoRa) wireless networks have been widely proposed as a effic...

Muti-Agent Proximal Policy Optimization For Data Freshness in UAV-assisted Networks

Unmanned aerial vehicles (UAVs) are seen as a promising technology to pe...

Cooperative Deep Reinforcement Learning for Multiple Groups NB-IoT Networks Optimization

NarrowBand-Internet of Things (NB-IoT) is an emerging cellular-based tec...

Joint Uplink-and-Downlink Optimization of 3D UAV Swarm Deployment for Wireless-Powered NB-IoT Networks

This paper investigates a full-duplex orthogonal-frequency-division mult...

On-board Deep Q-Network for UAV-assisted Online Power Transfer and Data Collection

Unmanned Aerial Vehicles (UAVs) with Microwave Power Transfer (MPT) capa...

Please sign up or login with your details

Forgot password? Click here to reset