AI-Augmented Data Engineering: Enhancing ETL Processes for Real-Time Analytics in Multi-Cloud Environments

Authors

  • Beverly DSouza, Karthik Puthraya, Venkata Sai Manoj Pasupuleti

Keywords:

AI-augmented ETL, multi-cloud environments, real-time analytics, Apache Kafka, TensorFlow, LSTM networks, data transformation, computational efficiency, scalability, statistical validation

Abstract

This study explores the transformative potential of AI-augmented ETL (Extract, Transform, Load) processes in enhancing real-time analytics for multi-cloud environments. By integrating advanced technologies such as Apache Kafka, TensorFlow, and LSTM networks, the proposed framework significantly improves efficiency, scalability, and accuracy compared to traditional ETL pipelines. Experimental results demonstrate a 47.3% reduction in latency, a transformation accuracy of 98.7%, and superior computational efficiency, with 20.9% lower CPU utilization and 73.5% higher GPU utilization. The framework's ability to handle heterogeneous data across AWS Redshift, Google BigQuery, and Azure SQL ensures seamless interoperability in multi-cloud architectures. Rigorous statistical analysis, including ANOVA and Pearson correlation, validates the framework's performance, while real-time analytics capabilities enable timely insights for applications such as financial forecasting and IoT-driven decision-making. This study highlights the critical role of AI in optimizing data engineering workflows, offering actionable insights for organizations seeking to leverage real-time analytics in distributed environments.

Downloads

Download data is not yet available.

References

Ahn, M. J., & Chen, Y. C. (2022). Digital transformation toward AI-augmented public administration: The perception of government employees and the willingness to use AI in government. Government Information Quarterly, 39(2), 101664.

Anand, C. R. (2022). Triangulation-Augmented AI-Algorithm for Achieving Intelligent Flight Stabilizing Performance Capabilities. In Intelligent Cyber-Physical Systems Security for Industry 4.0 (pp. 217-237). Chapman and Hall/CRC.

Arık, S. Ö., Shor, J., Sinha, R., Yoon, J., Ledsam, J. R., Le, L. T., ... & Pfister, T. (2021). A prospective evaluation of AI-augmented epidemiology to forecast COVID-19 in the USA and Japan. NPJ digital medicine, 4(1), 146.

Battina, D. S. (2016). AI-Augmented Automation for DevOps, a Model-Based Framework for Continuous Development in Cyber-Physical Systems. International Journal of Creative Research Thoughts (IJCRT), ISSN, 2320-2882.

Bergelin, J., & Strandberg, P. E. (2022, October). Industrial requirements for supporting ai-enhanced model-driven engineering. In Proceedings of the 25th international conference on model driven engineering languages and systems: companion proceedings (pp. 375-379).

Chen, L., Ye, M., Milne, A., Hillier, J., & Oglesby, F. (2022). The scope for AI-augmented interpretation of building blueprints in commercial and industrial property insurance. arXiv preprint arXiv:2205.01671.

Eramo, R., Muttillo, V., Berardinelli, L., Bruneliere, H., Gomez, A., Bagnato, A., ... & Cicchetti, A. (2021, September). Aidoart: Ai-augmented automation for devops, a model-based framework for continuous development in cyber-physical systems. In 2021 24th Euromicro Conference on Digital System Design (DSD) (pp. 303-310). IEEE.

Fradkin, L., Uskuplu Altinbasak, S., & Darmon, M. (2021). Towards explainable augmented intelligence (AI) for crack characterization. Applied Sciences, 11(22), 10867.

Ghafghazi, S., Carnett, A., Neely, L., Das, A., & Rad, P. (2021). AI-augmented behavior analysis for children with developmental disabilities: building toward precision treatment. IEEE Systems, Man, and Cybernetics Magazine, 7(4), 4-12.

Iftikhar, S., Raj, U., Tuli, S., Golec, M., Chowdhury, D., Gill, S. S., & Uhlig, S. (2022, December). TESCO: Multiple simulations based AI-augmented Fog computing for QoS optimization. In 2022 IEEE Smartworld, Ubiquitous Intelligence & Computing, Scalable Computing & Communications, Digital Twin, Privacy Computing, Metaverse, Autonomous & Trusted Vehicles (SmartWorld/UIC/ScalCom/DigitalTwin/PriComp/Meta) (pp. 2092-2099). IEEE.

Klein, M., Carpentier, T., Jeanclaude, E., Kassab, R., Varelas, K., de Bruijn, N., ... & Aligne, F. (2020, September). AI-augmented multi function radar engineering with digital twin: Towards proactivity. In 2020 IEEE Radar Conference (RadarConf20) (pp. 1-6). IEEE.

Kuhn, S. V., Bischof, R., Klonaris, G., Kaufmann, W., & Kraus, M. A. (2022). ntab0: Design priors for AI-augmented generative design of network tied-arch-bridges. In Proceedings of 33. Forum Bauinformatik (pp. 437-444). Technische Universität München.

Kwon, J. (2022). Machine Learning for AI-Augmented Design Space Exploration of Computer Systems. Columbia University.

Lee, J., Kundu, P., & Gore, P. (2022). Industrial AI-augmented Data-Centric Metrology for Highly Connected Production Systems. SMART AND NETWORKED MANUFACTURING, 20.

Palladino, A., Duff, M., Bock, A., Parsons, T., Arantes, R., Chartier, B., ... & Moore, K. (2021). Ai-augmented human performance evaluation for automated training decision support. In Intelligent Human Systems Integration 2021: Proceedings of the 4th International Conference on Intelligent Human Systems Integration (IHSI 2021): Integrating People and Intelligent Systems, February 22-24, 2021, Palermo, Italy (pp. 469-475). Springer International Publishing.

Pham, P., Nguyen, V., & Nguyen, T. (2022, October). A review of ai-augmented end-to-end test automation tools. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (pp. 1-4).

Toulkeridou, V. (2019). Steps towards AI augmented parametric modeling systems for supporting design exploration. Blucher Design Proceedings, 37, 81-92.

Zhou, C., Li, Y., Wu, C., Shao, Y., & Zhou, Y. (2021, December). Study of cognitive activity in AI-augmented design iteration support. In 2021 14th International Symposium on Computational Intelligence and Design (ISCID) (pp. 78-82). IEEE.

Downloads

Published

06.08.2024

How to Cite

Beverly DSouza. (2024). AI-Augmented Data Engineering: Enhancing ETL Processes for Real-Time Analytics in Multi-Cloud Environments. International Journal of Intelligent Systems and Applications in Engineering, 12(23s), 2475–2482. Retrieved from https://ijisae.org/index.php/IJISAE/article/view/7372

Issue

Vol. 12 No. 23s (2024)

Section

Research Article

License

Creative Commons License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.

IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.