A Constrained Partially Observable Markov Decision Process Framework for Optimizing Device-to-Device Communications in Cellular Networks


  • Manjula G. Associate Professor, Dept of CSE, BGS College of Engineering and Technology, Bengaluru, Karnataka, India
  • Nirmala J. Saunshimath Assistant professor, Nitte Meenakshi institute of technology, Karnataka India
  • Vinay T. R. Assistant Professor, Artificial intelligence and Data Science, Ramaiah Institute of Technology, Bengaluru, Karnataka, India
  • Pratibha Deshmukh University of Mumbai, Bharati Vidyapeeth’s Institute of Management and Information Technology, Navi Mumbai, Maharashtra, India
  • Sudhanshu Maurya Associate Professor, Symbiosis Institute of Technology, Nagpur Campus, India Symbiosis International (Deemed University), Pune, India
  • Pavithra G. Associate Professor, Dept. of Electronics & Communication Engineering, Dayananda Sagar College of Engineering (DSCE), Bangalore, Karnataka, India




This article proposes a constrained partially observable Markov decision process (CPOMDP) framework to model the decision-making problem of a group of low-battery cellular users trying to switch to device-to-device (D2D) mode while keeping a minimal distance between them. The CPOMDP defines the state space as the collective state of all users and the D2D mode, the observation space as the battery levels of the users, and the action space as the decision to transition to D2D mode or not. As a function of the state and action, the minimal distance constraints between users are included. The Bellman equation, the observation update equation, the belief update equation, and the policy update equation are among the equations satisfying the CPOMDP framework. The equations are modified to incorporate distance constraints as a penalty term within the reward function. The proposed framework can be utilised to offer users an optimal policy for transitioning to D2D mode while minimising the penalty for violating the distance constraint. The proposed framework can have substantial effects on cellular network resource efficiency, battery life improvement, and network congestion reduction.


Download data is not yet available.


Karina Valdivia Delgado; Scott Sanner; Leliane Nunes de Barros; Fábio Gagliardi Cozman; "Efficient Solutions to Factored MDPs with Imprecise Transition Probabilities", ARTIF. INTELL., 2009. (IF: 3)

] Aditya Undurti; Jonathan P. How; "An Online Algorithm for Constrained POMDPs", 2010 IEEE INTERNATIONAL CONFERENCE ON ROBOTICS AND ..., 2010. (IF: 3)

S] Lauren B. Davis; Russell E. King; Thom J. Hodgson; Wenbin Wei; "Information Sharing in Capacity Constrained Supply Chains Under Lost Sales", INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH, 2011. (IF: 3)

] Feng Wu; Nicholas R. Jennings; Xiaoping Chen; "Sample-Based Policy Iteration for Constrained DEC-POMDPs", 2012

Junhyuk Kim; Peng Yong Kong; Nah-Oak Song; June-Koo Kevin Rhee; Saleh R. Al-Araji; "MDP Based Dynamic Base Station Management for Power Conservation in Self-organizing Networks", 2014

FERENCE ..., 2014

Pedro Henrique Rodrigues Quemel e Assis Santana; Sylvie Thiébaux; Brian Williams; "RAO*: An Algorithm For Chance-Constrained POMDP’s", AAAI, 2016.

] Erwin Walraven; Matthijs T. J. Spaan; "Column Generation Algorithms for Constrained POMDPs", J. ARTIF. INTELL. RES., 2018

Yue Wang; Swarat Chaudhuri; Lydia E. Kavraki; "Bounded Policy Synthesis For POMDPs With Safe-Reachability Objectives", ARXIVCS.RO, 2018.

Michael C. Fowler; T. Charles Clancy; Ryan K. Williams; "Intelligent Knowledge Distribution: Constrained-Action POMDPs For Resource-Aware Multi-Agent Communication"

Michael C Fowler; T Charles Clancy; Ryan K Williams; "Intelligent Knowledge Distribution: Constrained-Action POMDPs for Resource-

Aware Multiagent Communication", IEEE TRANSACTIONS ON CYBERNETICS, 2022

Ali Hassan; Robert Mieth; Michael Chertkov; Deepjyoti Deka; Yury Dvorkin; "Optimal Load Ensemble Control In Chance-Constrained Optimal Power Flow", ARXIV-CS.SY, 2018

Mohammadhosein Hasanbeig; Alessandro Abate; Daniel Kroening; "Logically-Constrained Neural Fitted Q-Iteration", ARXIV-CS.LG, 2018.

Jongmin Lee; Geon-hyeong Kim; Pascal Poupart; Kee-Eung Kim; "Monte-Carlo Tree Search for Constrained POMDPs", NIPS, 2018

Mahmoud El Chamie; Yue Yu; Behçet Açıkmes¸e; Masahiro Ono; "Controlled Markov Processes With Safety State Constraints", IEEE TRANSACTIONS ON AUTOMATIC CONTROL, 2019.

Eleanor Quint; Dong Xu; Samuel Flint; Stephen Scott; Matthew Dwyer; "Formal Language Constraints For Markov Decision Processes", ARXIV-CS.LG, 2019.

Chunxia Su; Fang Ye; Li-Chun Wang; Li Wang; Yuan Tian; Zhu Han; "UAV-Assisted Wireless Charging for Energy-Constrained IoT Devices Using Dynamic Matching", IEEE INTERNET OF THINGS JOURNAL, 2020.

Jae-Mo Kang; "Reinforcement Learning Based Adaptive Resource Allocation for Wireless Powered Communication Systems", IEEE COMMUNICATIONS LETTERS, 2020.

Jie Li; Yong Zhou; He Chen; Yuanming Shi; "Age of Aggregated Information: Timely Status Update with Over-the-Air Computation", GLOBECOM 2020 - 2020 IEEE GLOBAL COMMUNICATIONS CONFERENCE, 2020.

Gabriel Kalweit; Maria Huegle; Moritz Werling; Joschka Boedecker; "Deep Constrained Q-learning", ARXIV-CS.LG, 2020.

Hannes Eriksson; Debabrota Basu; Tommy Tram; Mina Alibeigi; Christos Dimitrakakis; "Reinforcement Learning in The Wild with Maximum Likelihood-based Model Transfer", ARXIV-CS.LG, 202

Vikhyath K B and Achyutha Prasad N (2023), Optimal Cluster Head Selection in Wireless Sensor Network via Multi-constraint Basis using Hybrid Optimization Algorithm: NMJSOA. IJEER 11(4), 1087-1096. DOI: 10.37391/ijeer.110428.




How to Cite

G., M. ., Saunshimath, N. J. ., T. R., V. ., Deshmukh, P. ., Maurya, S. ., & G., P. . (2024). A Constrained Partially Observable Markov Decision Process Framework for Optimizing Device-to-Device Communications in Cellular Networks. International Journal of Intelligent Systems and Applications in Engineering, 12(20s), 487–494. Retrieved from https://ijisae.org/index.php/IJISAE/article/view/5161



Research Article

Most read articles by the same author(s)