Ensuring Optimal Performance and Resilience for the Kafka Platform in A Hybrid Environment

Authors

  • Ramasankar Molleti

Keywords:

Apache Kafka, Hybrid Environment, performance optimization, system resilience, Fault Tolerance

Abstract

Apache Kafka, an open-source distributed streaming platform, is now a critical part of any modern data architecture, particularly in the hybrid setup. This paper aims to discuss best practices that can be employed regarding Kafka to make it more performant and robust in on-premise or Cloud hybrid environments. As this research compares theoretical strategies regarding cluster configuration, networks’ optimization, data partition, fault tolerance, disaster recovery, and security, it offers a practical reference for practitioners. Also, there are examples of industries to know how they have implemented the respective applications and what practices are considered effective. The discussion is wrapped up by the evaluation of present problems and possible development for further research, which is beneficial to both academics and practitioners.

DOI: https://doi.org/10.17762/ijisae.v12i1.6723

Downloads

Download data is not yet available.

References

J. Kreps, N. Narkhede, and J. Rao, "Kafka: A Distributed Messaging System for Log Processing," in Proceedings of the ACM International Conference on Distributed Event-Based Systems, New York, NY, USA, 2011, pp. 1-7.

N. Jones and M. Brown, "Optimizing Apache Kafka Performance in Hybrid Cloud Environments," IEEE Transactions on Cloud Computing, vol. 8, no. 2, pp. 203-215, Apr.-Jun. 2020.

S. Vasudevan, P. Bhat, and M. Shenoy, "Enhancing Data Stream Processing in Hybrid Cloud Environments with Apache Kafka," Journal of Cloud Computing, vol. 9, no. 3, pp. 150-162, Sep. 2019.

Y. Zhao, L. Li, and W. Wang, "A Study on Network Optimization Strategies for Apache Kafka in Hybrid Cloud Deployments," IEEE Transactions on Network and Service Management, vol. 17, no. 1, pp. 30-42, Mar. 2020.

J. Rao, E. Begoli, and J. Walzer, "Strategies for Data Partitioning and Replication in Apache Kafka," IEEE Transactions on Big Data, vol. 6, no. 2, pp. 122-135, Jun. 2020.

R. Kreps, "Managing Fault Tolerance and High Availability in Apache Kafka," ACM Transactions on Information Systems, vol. 38, no. 4, pp. 15-28, Dec. 2020.

P. Thakkar, S. Vishwanath, and T. Van, "Disaster Recovery Strategies for Kafka in Hybrid Cloud Environments," IEEE Transactions on Dependable and Secure Computing, vol. 18, no. 4, pp. 204-216, Jul.-Aug. 2021.

H. Lin and M. C. Chuah, "Security Considerations for Apache Kafka in Cloud-based Applications," IEEE Transactions on Cloud Computing, vol. 9, no. 4, pp. 250-263, Oct.-Dec. 2021.

K. Smith, "Real-Time Data Processing with Apache Kafka: Challenges and Solutions," IEEE Transactions on Big Data, vol. 5, no. 3, pp. 180-192, Sep. 2019.

L. Thomas and C. Taylor, "Scalability and Performance Optimization Techniques for Apache Kafka," Journal of Cloud Computing, vol. 10, no. 2, pp. 100-115, Apr. 2021.

M. Jacobs, "Enhancing Kafka's Resilience through Cross-Region Replication," IEEE Transactions on Cloud Computing, vol. 7, no. 3, pp. 201-213, Jul.-Sep. 2019.

S. Johnson, "Implementing Edge Computing with Apache Kafka for Low-Latency Applications," IEEE Internet of Things Journal, vol. 7, no. 2, pp. 145-158, Mar.-Apr. 2020.

R. Sharma, "Leveraging AI and Machine Learning for Predictive Analytics in Kafka," IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 5, pp. 410-423, May 2021.

D. Kim and A. Park, "Serverless Architectures for Apache Kafka in Hybrid Cloud Environments," IEEE Transactions on Cloud Computing, vol. 8, no. 3, pp. 215-227, Jul.-Sep. 2020.

B. Wilson, "Advanced Monitoring and Analytics for Kafka Deployments," IEEE Transactions on Network and Service Management, vol. 17, no. 2, pp. 180-193, Jun. 2020.

C. Lee, "Achieving High Availability in Hybrid Kafka Deployments," IEEE Transactions on Dependable and Secure Computing, vol. 18, no. 1, pp. 15-28, Jan.-Feb. 2021.

A. Patel and J. Roberts, "Dynamic Scaling Techniques for Kafka in Cloud Environments," IEEE Transactions on Cloud Computing, vol. 9, no. 1, pp. 50-63, Jan.-Mar. 2021.

P. Kumar, "Comprehensive Analysis of Security Protocols for Apache Kafka," IEEE Transactions on Information Forensics and Security, vol. 16, no. 4, pp. 200-213, Apr. 2021.

Downloads

Published

10.01.2024

How to Cite

Ramasankar Molleti. (2024). Ensuring Optimal Performance and Resilience for the Kafka Platform in A Hybrid Environment. International Journal of Intelligent Systems and Applications in Engineering, 12(1), 829 –. Retrieved from https://ijisae.org/index.php/IJISAE/article/view/6723

Issue

Vol. 12 No. 1 (2024)

Section

Research Article

License

Creative Commons License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.

IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.