Resilient UAV Deployment and User Association Optimization in Multi-RAT HetNets using DRL

Andrew Habila John, Sokyes Armak Lapak, Agbon E. E., Umar Abubakar, Aliyu Umar Abubakar

Abstract


This study proposes a Resilient Multi-Agent Deep Deterministic Policy Gradient (R-MADDPG) framework for UAV-assisted Heterogeneous Networks (HetNets) operating under realistic communication constraints. The proposed framework addresses three critical challenges in existing approaches: vulnerability to communication outages, inaccurate energy modeling, and the performance gap between centralized training and decentralized deployment. R-MADDPG introduces three key innovations: (i) communication dropout training with 30% dropout probability to develop policies resilient to partial observability, (ii) an enhanced energy model that separates idle and active power consumption across multiple radio access technologies (LTE, Wi-Fi, control links), and (iii) a hybrid architecture with centralized critics enabling near-optimal decentralized execution. Extensive simulations compare R-MADDPG against DDPG and DDQN variants under both centralized and decentralized training paradigms across varying network densities from 60 to 100 ground devices. Results demonstrate that R-MADDPG achieves average SER improvements of 2.0% in centralized scenarios and 3.5% in decentralized scenarios compared to the best existing methods, with all satisfaction-to-energy ratio (SER) values maintained between 0.85 and 0.99. Under communication outages of 5-60 seconds, the existing framework collapses to 10-60% of baseline performance, while R-MADDPG maintains 65-95% performance, representing a 35-70 percentage point improvement. The enhanced energy model reduces prediction error by 88.6% compared to conventional approaches, from 16.6% to just 1.9% mean absolute error. Furthermore, R-MADDPG exhibits remarkable robustness to decentralization, with only 0.4% performance degradation from centralized to decentralized deployment compared to 2.6-3.2% for existing methods. These results validate R-MADDPG as an effective solution for reliable UAV-assisted communication in challenging environments with limited or unreliable infrastructure. 


Full Text:

PDF

References


Al-Hourani, A., Kandeepan, S., & Lardner, S. (2014). Optimal LAP altitude for maximum coverage. IEEE Wireless Communications Letters, 3(6), 569–572.

Anany, M. G., Elmesalawy, M. M., Abd El Haleem, A. M., & Ibrahim, I. I. (2025). Deep reinforcement learning framework for joint optimization of multi-RAT UAV location and user association in heterogeneous networks. Scientific Reports, 15, 39013. https://doi.org/10.1038/s41598-025-22610-1

Boyd, S., & Vandenberghe, L. (2004). Convex optimization. Cambridge University Press.

Clerigues, D., Wubben, J., Calafate, C. T., & Cano, J. C. (2024). Enabling resilient UAV swarms through multi-hop wireless communications. EURASIP Journal on Wireless Communications and Networking, 39. https://doi.org/10.1186/s13638-024-02373-5

Feng, Z., Wu, D., Huang, M., & Yuen, C. (2024). Graph attention based reinforcement learning for trajectory design and resource assignment in multi UAV assisted communication. arXiv. https://arxiv.org/abs/2401.XXXXX

Gao, Y., Liu, M., Yuan, X., Hu, Y., & Sun, P. (2024). Federated deep reinforcement learning based trajectory design for UAV assisted networks with mobile ground devices. Scientific Reports, 14, 22753. https://doi.org/10.1038/s41598-024-22753-x

Gupta, L., Jain, R., & Vaszkun, G. (2016). Survey of important issues in UAV communication networks. IEEE Communications Surveys & Tutorials, 18(2), 1123–1152. https://doi.org/10.1109/COMST.2016.2511938

Kim, M., Lee, H., Hwang, S., Debbah, M., & Lee, I. (2024). Cooperative multi-agent deep reinforcement learning methods for UAV aided mobile edge computing networks. arXiv. https://arxiv.org/abs/2403.XXXXX

Liu, L., Zhang, H., Letaief, K. B., Chen, H., & Wang, J. (2016). User association in 5G networks: A survey and an outlook. IEEE Communications Surveys & Tutorials, 18(2), 1018–1044. https://doi.org/10.1109/COMST.2015.2476347

Lyu, J., Zeng, Y., Zhang, R., & Lim, T. J. (2017). Placement optimization of UAV-mounted mobile base stations. IEEE Communications Letters, 21(3), 604–607. https://doi.org/10.1109/LCOMM.2016.2641060

Maddah-Ali, M. A., & Niesen, U. (2014). Fundamental limits of caching. IEEE Transactions on Information Theory, 60(5), 2856–2867. https://doi.org/10.1109/TIT.2014.2317686

Mozaffari, M., Saad, W., Bennis, M., Nam, Y.-H., & Debbah, M. (2019). A tutorial on UAVs for wireless networks: Applications, challenges, and open problems. IEEE Communications Surveys & Tutorials, 21(3), 2334–2360. https://doi.org/10.1109/COMST.2019.2909032

Shoaib, M., Husnain, G., Khan, M., et al. (2025). Decentralized resource allocation in UAV communication networks through reward based multi agent learning. Scientific Reports, 15, 33122. https://doi.org/10.1038/s41598-025-18353-8

Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction (2nd ed.). MIT Press.

Tan, M. (1993). Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the Tenth International Conference on Machine Learning (pp. 330–337).

Zeng, Y., Zhang, R., & Lim, T. J. (2016). Wireless communications with unmanned aerial vehicles: Opportunities and challenges. IEEE Communications Magazine, 54(5), 36–42. https://doi.org/10.1109/MCOM.2016.7470933


Refbacks

  • There are currently no refbacks.