Hybrid MaAchine Learning for Detecting Faulty Nodes in Hadoop Environments
Keywords:
Hybrid Machine Learning, Hadoop, Fault Detection, Distributed Systems, Predictive Maintenance.Abstract
Hadoop, a widely adopted framework for distributed data processing, faces significant challenges related to node failures, which can lead to increased job failure rates and reduced system efficiency. Traditional monitoring and fault detection mechanisms often struggle to handle the dynamic nature of distributed systems, leading to prolonged downtime and inefficient resource utilization. This paper proposes a hybrid machine learning (ML) framework for the detection of faulty nodes within Hadoop clusters, utilizing system logs, CPU usage data, and latency metrics to predict potential node failures. By leveraging advanced predictive models and integrating corrective actions, this approach ensures improved fault tolerance, reduced job failures, and enhanced resource optimization. Experimental results demonstrate the effectiveness of the hybrid ML model in detecting faulty nodes early and mitigating the impact on the overall performance of Hadoop clusters.
Downloads
References
Wang, X., & Yang, Y. (2021). “Hybrid Models for Predictive Maintenance in Big Data Environments.” International Journal of Machine Learning, 22(3), 255-274.
Liu, F., et al. (2018). “Fault Detection and Diagnosis in Distributed Systems: A Machine Learning Approach.” Journal of Parallel and Distributed Computing, 121, 1-11.
Kaur, P., & Bansal, J. (2020). “Machine Learning Based Fault Detection in Hadoop Ecosystems.” International Journal of Computer Applications, 178(12), 28-35.
Tan, S., et al. (2017). “An Intelligent Fault Detection and Diagnosis System for Big Data Platforms.” Computers & Electrical Engineering, 59, 264-276.
Zhang, Y., & Zhang, L. (2020). “Data-Driven Approaches for Fault Prediction and Fault Tolerance in Hadoop Environments.” IEEE Transactions on Big Data, 6(2), 235-245.
Xie, J., et al. (2019). “Fault-Tolerant Techniques for Hadoop and MapReduce: A Survey.” Future Generation Computer Systems, 92, 145-156.
Du, J., et al. (2018). “A Hybrid Model for Fault Detection and Prognosis in Distributed Systems.” IEEE Transactions on Network and Service Management, 15(1), 10-23.
Ouyang, Q., et al. (2018). “Using Machine Learning Techniques for Fault Detection in Cloud Environments.” International Journal of Cloud Computing and Services Science, 7(2), 45-59.
Liao, Z., et al. (2019). “Predictive Maintenance Using Machine Learning for Hadoop Clusters.” International Journal of Advanced Computer Science and Applications, 10(6), 126-134.
Jia, W., & Sun, Y. (2020). “A Comprehensive Review of Fault Detection Algorithms for Hadoop.” Journal of Computer Science and Technology, 35(2), 215-230.
Kumar, R., & Singh, M. (2019). “Fault Tolerance in Big Data Systems Using Machine Learning Algorithms.” Computer Applications in Engineering Education, 27(3), 798-807.
Yadav, V., et al. (2021). “Predictive Analytics for Fault Detection in Hadoop Distributed Systems Using Ensemble Learning.” Proceedings of the International Conference on Big Data Analytics, 45-52.
Lee, Y., et al. (2017). “Enhancing Fault Detection with Hybrid Deep Learning Models in Hadoop.” Journal of Information and Computational Science, 14(9), 1351-1361.
Sharma, P., & Gupta, D. (2020). “Machine Learning for Fault Detection and Prediction in Big Data Platforms.” Advanced Computing Technologies, 9(4), 290-304.
Rao, S., et al. (2019). “A Survey on Fault Tolerance and Error Recovery in Hadoop Ecosystem.” International Journal of Computer Applications, 177(4), 12-20.
Xu, Z., & Liu, Y. (2020). “Fault Prediction in Hadoop Distributed Systems Using Time Series Forecasting Techniques.” IEEE Transactions on Cloud Computing, 8(7), 1800-1810.
Chen, S., et al. (2020). “Hybrid Fault Detection Models for Distributed Computing Systems: A Comparative Study.” International Journal of Cloud Computing and Services Science, 8(5), 1-10.
Chen, Y., & Zhang, X. (2018). “Hadoop Fault Detection Using Decision Trees: A Machine Learning Approach.” Computer Networks, 143, 35-45.
Xiao, J., et al. (2021). “Fault Prediction in Big Data Systems: The Role of Hybrid Machine Learning Models.” International Journal of Computer Science and Technology, 29(3), 450-461.
Li, X., et al. (2020). “Leveraging Hybrid Machine Learning for Predictive Maintenance in Hadoop Ecosystems.” Journal of Parallel and Distributed Computing, 134, 43-52.
Verma, A., & Sharma, K. (2019). “Enhanced Fault Detection in Hadoop Using Random Forest and Neural Networks.” Journal of Computing and Security, 12(1), 105-115.
Tiwari, P., & Mehta, S. (2020). “A Hybrid Machine Learning Approach for Fault Detection in Big Data Systems.” IEEE Transactions on Big Data, 6(1), 73-85.
Patel, M., & Dutta, P. (2021). “Hybrid ML-Based Fault Prediction for Hadoop Distributed Systems: A Comprehensive Review.” International Journal of Machine Learning and Data Mining, 10(2), 56-68.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.
IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
