Optimizing Resource Allocation in Cloud Systems using Reinforcement Learning Driven Dynamic VM Placement
Keywords:
Dynamic VM Allocation, Reinforcement Learning, Experience Replay, Optimization, A3C algorithmAbstract
Virtual machines (VMs) are extensively used these days as a substitute for physical machines. When the computation power requirement goes beyond that of the existing physical systems, based on client-specific memory requirements, their tools subscriptions, and services the appropriate number of VMs needs to be allocated dynamically. The aim is to minimize the resource cost and energy consumption for optimal usage and enhancement of savings. This is hence an optimization problem that needs to be addressed based on various parameters linked to the system. In this paper, we have worked towards the allocation or placement of VMs in a cloud system, where based on previous requirements we train a model by reinforcement using the A3C algorithm, considering the replays of experiences in various states of the environment to ensure optimal allocation of VMs and hence the real-time functionality of the cloud system.
Downloads
References
U. C. De, R. Satapathy & S. S. Patra, "Optimizing Resource Allocation using Proactive Predictive Analytics and ML-Driven Dynamic VM Placement," 4th Global Conference for Advancement in Technology, 2023.
U. C. De, R. Satapathy and S. S. Patra, "Cost Analysis and Optimization of Virtual Machine Allocation in the Cloud Data Center," International Conference on Inventive Computation Technologies (ICICT), pp. 809-813, IEEE, 2023.
S. M. Seyyedsalehi and M. Khansari, "Virtual Machine Placement Optimization for Big Data Applications in Cloud Computing," in IEEE Access, 10, pp. 96112-96127, 2022.
S. Mejahed, M. Elshrkawey, "A multi-objective algorithm for virtual machine placement in cloud environments using a hybrid of particle swarm optimization and flower pollination optimization", PeerJ Computer Science, 8, e834, 2022.
Sudhakar, Saravanan, "A Survey and Future Studies of Virtual Machine Placement Approaches in Cloud Computing Environment", 6th International Conference on Cloud Computing and Internet of Things, pp. 15-21, 2021.
A. Alashaikh, E. Alanazi, A. Al-Fuqaha, "A survey on the use of preferences for virtual machine placement in cloud data centers", ACM Computing Surveys (CSUR), 54(5), pp. 1-39, 2021.
W. Zhang, X. Chen, J. Jiang, "A multi-objective optimization method of initial virtual machine fault-tolerant placement for star topological data centers of cloud systems", Tsinghua Science and Technology, 26(1), pp. 95-111, 2020.
Farzaneh Abazari, Morteza Analoui, Hassan Takabi, Song Fu, "MOWS: Multi-objective workflow scheduling in cloud computing based on heuristic algorithm", Simulation Modelling Practice and Theory, 93, pp. 119-132,2019.
M. C. Silva Filho, C. C. Monteiro, P. R. Inácio, M. M. Freire, "Approaches for optimizing virtual machine placement and migration in cloud environments: A survey", Journal of Parallel and Distributed Computing, 111, pp. 222-250, 2018.
Ao Zhou, Shangguang Wang, Member, Bo Cheng, Member, Zibin Zheng, Member, Fangchun Yang, Senior Member, Rong N. Chang, Senior Member, Michael R. Lyu, Fellow, Rajkumar Buyya, "Cloud Service Reliability Enhancement via Virtual Machine Placement Optimization”, IEEE Transactions on Services Computing, 10(6), pp. 902-913, 2017.
J. Gao, G. Tang, "Virtual Machine Placement Strategy Research", International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery, 2013, pp. 294-297,2013.
B. B. Dash, R. Satapathy, S. S. Patra, "SDN-Assisted Routing Scheme in Cloud Data Center using Queueing Vacation Policy", 2nd International Conference on Edge Computing and Applications (ICECAA), pp. 1-6, 2023.
B. B. Dash, R. Satapathy and S. S. Patra, "Energy Efficient SDN-assisted Routing Scheme in Cloud Data Center", 2nd International Conference on Vision Towards Emerging Trends in Communication and Networking Technologies (ViTECoN), pp. 1-5, 2023.
S. Behera, N. Panda, U. C. De, B. B. Dash, B. Dash, S. S. Patra, "A task offloading scheme with Queue Dependent VM in fog Center”, 6th International Conference on Information Systems and Computer Networks (ISCON), pp. 1-5, 2023.
B. B. Dash, S. S. Patra, R. Satpathy and B. Dash, "Improvement of SDN-based Task Offloading using Golden Jackal Optimization in Fog Center," World Conference on Communication & Computing (WCONF), pp. 1-6,2023.
S. S. Patra, R. Govindaraj, S. Chowdhury, M. A. Shah, R. Patro and S. Rout, "Energy Efficient End Device Aware Solution Through SDN in Edge-Cloud Platform," in IEEE Access, vol. 10, pp. 115192-115204, 2022.
Inès De Courchelle, Tom Guérout, Georges Da Costa, Thierry Monteil, Yann Labit, “Green energy efficient scheduling management”, Simulation Modelling Practice and Theory, Volume 93, pp. 208-232, 2019.
M. Kaloev, G. Krastev, "Experiments Focused on Exploration in Deep Reinforcement Learning", 5th International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT), pp. 351-355, 2021.
J. Kiefer and K. Dorer, "Double Deep Reinforcement Learning", 2023 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC), pp. 17-22, 2023.
M. H. Krishna and M. M. Latha, "Complexity and Performance Evaluation of Segmented and Recursive Reinforcement Learning", IEEE 4th International Conference on Computing, Power and Communication Technologies (GUCON), pp. 1-7, 2021.
Xiuyuan Lu, Benjamin Van Roy, Vikranth Dwaracherla, Morteza Ibrahimi, Ian Osband, Zheng Wen, "Reinforcement Learning Bit by Bit", now, 2023.
Thomas M. Moerland, Joost Broekens, Aske Plaat and Catholijn M. Jonker, "Model-based Reinforcement Learning: A Survey", Foundations and Trends® in Machine Learning, 16(1), pp 1-118, 2023.
Kyriakos G. Vamvoudakis and Nick-Marios T. Kokolakis, "Synchronous Reinforcement Learning-Based Control for Cognitive Autonomy", Foundations and Trends® in Systems and Control, 8(1–2), pp 1-175, 2020.
Frank L. Lewis, Derong Liu, "Reinforcement Learning Control with Time-Dependent Agent Dynamics," in Reinforcement Learning and Approximate Dynamic Programming for Feedback Control, IEEE, pp.203-220, 2013.
Frank L. Lewis, Derong Liu, "An Actor-Critic-Identifier Architecture for Adaptive Approximate Optimal Control", in Reinforcement Learning and Approximate Dynamic Programming for Feedback Control, IEEE, pp.258-280, 2013.
T. Swain, M. Rath, J. Mishra, S. Banerjee and T. Samant, "Deep Reinforcement Learning based Target Detection for Unmanned Aerial Vehicle", IEEE India Council International Subsections Conference (INDISCON), pp. 1-5, 2022.
S. Banerjee, T. Swain, J. Mishra, M. K. Rath and T. Samant, "Surveillance using Unmanned Aerial Vehicle for Triggered Activity Capturing," 1st IEEE International Conference on Industrial Electronics: Developments & Applications (ICIDeA), pp. 6-11, 2022.
J. Luo, C. Paduraru, O. Voicu, Y. Chervonyi, S. Munns, J. Li, ... & D. J. Mankowitz, "Controlling commercial cooling systems using reinforcement learning." arXiv preprint arXiv:2211.07357, 2022.
R. S. Sutton, A. G. Barto, "Reinforcement learning: An introduction", MIT press, 2018.
R. Bellman, "The theory of dynamic programming", Bulletin of the American Mathematical Society, 60(6), pp. 503-515, 1954.
D. J. White, "A survey of applications of Markov decision processes", Journal of the operational research society, 44(11), pp. 1073-1096, 1993.
Martijn Van Otterlo, "Markov decision processes: Concepts and algorithms." Course on Learning and Reasoning, 2009.
Richard S. Sutton, "Learning to predict by the methods of temporal differences", Machine learning, 3(9-44),1988.
T. Schaul, J. Quan, I. Antonoglou, D. Silver, "Prioritized experience replay", arXiv preprint arXiv:1511.05952, 2015.
M. Tokic, "Adaptive ε-greedy exploration in reinforcement learning based on value differences", In Annual Conference on Artificial Intelligence, pp. 203-210, 2010.
M. Babaeizadeh, I. Frosio, S. Tyree, J. Clemons, & J. Kautz, "Reinforcement Learning through Asynchronous Advantage Actor-Critic on a GPU", ArXiv. /abs/1611.06256, 2016.
J. Schulman, P. Moritz, S. Levine, M. Jordan, & P. Abbeel, "High-dimensional continuous control using generalized advantage estimation", arXiv preprint arXiv:1506.02438, 2015.
V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, & M. Riedmiller, "Playing Atari with Deep Reinforcement Learning", ArXiv. /abs/1312.5602, 2013.
R. J. Williams, & J. Peng, "Function optimization using connectionist reinforcement learning algorithms", Connection Science, 3(3), pp. 241-268, 1991.
A. Verma, L. Pedrosa, M. Korupolu, D. Oppenheimer, E. Tune, J. Wilkes, "Large-scale cluster management at Google with Borg", In Proceedings of the tenth european conference on computer systems, pp. 1-17, 2015.
M. Tirmazi, A. Barker, N. Deng, M. E. Haque, Z. G. Qin, S. Hand, J. Wilkes, "Borg: the next generation", In Proceedings of the fifteenth European conference on computer systems, pp. 1-14, 2020.
Victor Chudnovsky, Rasekh Rifaat, Joseph Hellerstein, Bikash Sharma, Chita Das, “Modeling and Synthesizing Task Placement Constraints in Google Compute Clusters”, Symposium on Cloud Computing (SoCC), 2011.
M. Carvalho,W. Cirne, F. Brasileiro, J. Wilkes, "Long-term SLOs for reclaimed cloud computing resources", in Proceedings of the ACM Symposium on Cloud Computing, pp. 1-13, 2014.
A. Nair, P. Srinivasan, S.Blackwell, C. Alcicek, R. Fearon, , A. De Maria, ..., D. Silver, "Massively parallel methods for deep reinforcement learning", arXiv preprint arXiv:1507.04296, 2015.
Z. Wang, T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, N. Freitas, "Dueling network architectures for deep reinforcement learning", in International conference on machine learning, pp. 1995-2003, 2016.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.
IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
