A Constrained Partially Observable Markov Decision Process Framework for Optimizing Device-to-Device Communications in Cellular Networks
Keywords:
MDP, CMDP, POMDP, D2DAbstract
This article proposes a constrained partially observable Markov decision process (CPOMDP) framework to model the decision-making problem of a group of low-battery cellular users trying to switch to device-to-device (D2D) mode while keeping a minimal distance between them. The CPOMDP defines the state space as the collective state of all users and the D2D mode, the observation space as the battery levels of the users, and the action space as the decision to transition to D2D mode or not. As a function of the state and action, the minimal distance constraints between users are included. The Bellman equation, the observation update equation, the belief update equation, and the policy update equation are among the equations satisfying the CPOMDP framework. The equations are modified to incorporate distance constraints as a penalty term within the reward function. The proposed framework can be utilised to offer users an optimal policy for transitioning to D2D mode while minimising the penalty for violating the distance constraint. The proposed framework can have substantial effects on cellular network resource efficiency, battery life improvement, and network congestion reduction.
Downloads
References
Karina Valdivia Delgado; Scott Sanner; Leliane Nunes de Barros; Fábio Gagliardi Cozman; "Efficient Solutions to Factored MDPs with Imprecise Transition Probabilities", ARTIF. INTELL., 2009. (IF: 3)
] Aditya Undurti; Jonathan P. How; "An Online Algorithm for Constrained POMDPs", 2010 IEEE INTERNATIONAL CONFERENCE ON ROBOTICS AND ..., 2010. (IF: 3)
S] Lauren B. Davis; Russell E. King; Thom J. Hodgson; Wenbin Wei; "Information Sharing in Capacity Constrained Supply Chains Under Lost Sales", INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH, 2011. (IF: 3)
] Feng Wu; Nicholas R. Jennings; Xiaoping Chen; "Sample-Based Policy Iteration for Constrained DEC-POMDPs", 2012
Junhyuk Kim; Peng Yong Kong; Nah-Oak Song; June-Koo Kevin Rhee; Saleh R. Al-Araji; "MDP Based Dynamic Base Station Management for Power Conservation in Self-organizing Networks", 2014
FERENCE ..., 2014
Pedro Henrique Rodrigues Quemel e Assis Santana; Sylvie Thiébaux; Brian Williams; "RAO*: An Algorithm For Chance-Constrained POMDP’s", AAAI, 2016.
] Erwin Walraven; Matthijs T. J. Spaan; "Column Generation Algorithms for Constrained POMDPs", J. ARTIF. INTELL. RES., 2018
Yue Wang; Swarat Chaudhuri; Lydia E. Kavraki; "Bounded Policy Synthesis For POMDPs With Safe-Reachability Objectives", ARXIVCS.RO, 2018.
Michael C. Fowler; T. Charles Clancy; Ryan K. Williams; "Intelligent Knowledge Distribution: Constrained-Action POMDPs For Resource-Aware Multi-Agent Communication"
Michael C Fowler; T Charles Clancy; Ryan K Williams; "Intelligent Knowledge Distribution: Constrained-Action POMDPs for Resource-
Aware Multiagent Communication", IEEE TRANSACTIONS ON CYBERNETICS, 2022
Ali Hassan; Robert Mieth; Michael Chertkov; Deepjyoti Deka; Yury Dvorkin; "Optimal Load Ensemble Control In Chance-Constrained Optimal Power Flow", ARXIV-CS.SY, 2018
Mohammadhosein Hasanbeig; Alessandro Abate; Daniel Kroening; "Logically-Constrained Neural Fitted Q-Iteration", ARXIV-CS.LG, 2018.
Jongmin Lee; Geon-hyeong Kim; Pascal Poupart; Kee-Eung Kim; "Monte-Carlo Tree Search for Constrained POMDPs", NIPS, 2018
Mahmoud El Chamie; Yue Yu; Behçet Açıkmes¸e; Masahiro Ono; "Controlled Markov Processes With Safety State Constraints", IEEE TRANSACTIONS ON AUTOMATIC CONTROL, 2019.
Eleanor Quint; Dong Xu; Samuel Flint; Stephen Scott; Matthew Dwyer; "Formal Language Constraints For Markov Decision Processes", ARXIV-CS.LG, 2019.
Chunxia Su; Fang Ye; Li-Chun Wang; Li Wang; Yuan Tian; Zhu Han; "UAV-Assisted Wireless Charging for Energy-Constrained IoT Devices Using Dynamic Matching", IEEE INTERNET OF THINGS JOURNAL, 2020.
Jae-Mo Kang; "Reinforcement Learning Based Adaptive Resource Allocation for Wireless Powered Communication Systems", IEEE COMMUNICATIONS LETTERS, 2020.
Jie Li; Yong Zhou; He Chen; Yuanming Shi; "Age of Aggregated Information: Timely Status Update with Over-the-Air Computation", GLOBECOM 2020 - 2020 IEEE GLOBAL COMMUNICATIONS CONFERENCE, 2020.
Gabriel Kalweit; Maria Huegle; Moritz Werling; Joschka Boedecker; "Deep Constrained Q-learning", ARXIV-CS.LG, 2020.
Hannes Eriksson; Debabrota Basu; Tommy Tram; Mina Alibeigi; Christos Dimitrakakis; "Reinforcement Learning in The Wild with Maximum Likelihood-based Model Transfer", ARXIV-CS.LG, 202
Vikhyath K B and Achyutha Prasad N (2023), Optimal Cluster Head Selection in Wireless Sensor Network via Multi-constraint Basis using Hybrid Optimization Algorithm: NMJSOA. IJEER 11(4), 1087-1096. DOI: 10.37391/ijeer.110428.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.
IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
