Autonomous AI System for End-to-End Data Engineering
Keywords:
reinforcement, LLM, mechanisms, developmentAbstract
Autonomous data engineering is becoming increasingly essential for large-scale analytics, machine learning, and real-time enterprise decision systems, where continuous data availability and accuracy are mission-critical to operational success. Traditional enterprise data pipelines still rely heavily on manual configuration, hand-written transformations, and labor-intensive quality validation, resulting in slow development cycles, inconsistent data quality, and higher operational overhead (Rahm & Do, 2000; Stonebraker et al., 2017). Recent advances in large language models (LLMs) and automated data processing frameworks have opened new opportunities for intelligent, adaptive, and self-governing data systems (Devlin et al., 2019; OpenAI, 2023). In response to these challenges, this paper introduces AIDE-End, an Autonomous Artificial Intelligence System for End-to-End Data Engineering. The system executes the full data preparation lifecycle without human intervention by integrating transformer-based language models (Vaswani et al., 2017), reinforcement learning for corrective decision-making (Silver et al., 2017; Silver et al., 2020), metadata-driven intelligence, and policy-aware governance mechanisms. LLM-driven agents autonomously interpret schemas, detect anomalies, and generate executable transformation logic in SQL, Spark, and Python, extending capabilities demonstrated in AutoML and automated data transformation research (Chen & Weinberger, 2021; Lakshmanan et al., 2022). A metadata-centric layer maintains lineage, schema evolution, semantic relationships, and data quality metrics, aligning with modern data governance and lakehouse architectures (Armbrust et al., 2020; Databricks, 2022). To evaluate system performance, large datasets from financial transactions, healthcare claims, and e-commerce catalogs were analyzed. Experimental results demonstrate substantial performance gains over traditional ETL workflows, including faster execution, improved anomaly detection accuracy, and significant reductions in manual engineering effort—consistent with trends reported in automated data curation and self-managing pipelines (Hellerstein et al., 2012; Bernecker & Plattner, 2020). The system exhibits strong robustness against schema drift, inconsistent formats, and unstructured attributes, which are common failure points in manually designed pipelines. This research offers one of the first comprehensive demonstrations of a fully autonomous, AI-driven data engineering system capable of self-management from ingestion to deployment. By unifying LLM reasoning, reinforcement-learning optimization, and metadata-centric governance in a single autonomic framework, AIDE-End establishes a strong foundation for next-generation enterprise data platforms. The findings highlight significant improvements in analytics readiness, reduced operational burden, and increased trust in enterprise data ecosystems—directly supporting the emerging shift toward intelligent, self-maintaining data infrastructure.
DOI: https://doi.org/10.17762/ijisae.v12i13s.7964
Downloads
References
J. Dean, “The deep learning revolution and its implications,” Commun. ACM, vol. 62, no. 6, pp. 58–65, Jun. 2019.
T. Brown et al., “Language models are few-shot learners,” in Adv. Neural Inf. Process. Syst., vol. 33, pp. 1877–1901, 2020.
X. Chen and K. Q. Weinberger, “AutoML: A survey of the state-of-the-art,” ACM Comput. Surv., vol. 54, no. 8, pp. 1–36, Oct. 2021.s
Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, pp. 436–444, May 2015.
J. Kelleher, Data Science: Principles and Practice. MIT Press, 2020.
H. He, L. Deng, and A. Mohamed, “Deep learning for natural language processing,” IEEE Signal Process. Mag., vol. 34, no. 4, pp. 14–22, Jul. 2017.
A. Karpathy, “Software 2.0,” Medium, Nov. 2017. [Online]. Available: https://medium.com
M. Zaharia et al., “Apache Spark: Cluster computing with working sets,” in Proc. HotCloud, 2010.
M. Armbrust et al., “Delta Lake: High-performance ACID table storage over cloud object stores,” Proc. VLDB, vol. 13, no. 12, pp. 3411–3424, 2020.
M. Stonebraker et al., “Data curation at scale: The data civilizer system,” in Proc. CIDR, 2017.
T. Mikolov et al., “Efficient estimation of word representations in vector space,” arXiv:1301.3781, 2013.
J. Devlin et al., “BERT: Pre-training of deep bidirectional transformers,” in Proc. NAACL, 2019.
OpenAI, “GPT-4 technical report,” arXiv:2303.08774, 2023.
L. Floridi and M. Chiriatti, “GPT-3: Its nature, scope, limits, and consequences,” Minds Mach., vol. 30, no. 4, pp. 681–694, 2020.
J. Dean and S. Ghemawat, “MapReduce: Simplified data processing on large clusters,” Commun. ACM, vol. 51, no. 1, pp. 107–113, 2008.
F. Pedregosa et al., “Scikit-learn: Machine learning in Python,” J. Mach. Learn. Res., vol. 12, pp. 2825–2830, 2011.
K. He et al., “Deep residual learning for image recognition,” in Proc. CVPR, 2016.
D. Silver et al., “Mastering the game of Go with deep neural networks and tree search,” Nature, vol. 529, pp. 484–489, 2016.
D. Silver et al., “Reinforcement learning: A survey,” Found. Trends Mach. Learn., vol. 15, no. 1, pp. 1–140, 2020.
C. Sutton and A. McCallum, “An introduction to conditional random fields,” Found. Trends Mach. Learn., vol. 4, no. 4, pp. 267–373, 2012.
E. Rahm and H. H. Do, “Data cleaning: Problems and current approaches,” IEEE Data Eng. Bull., vol. 23, no. 4, pp. 3–13, 2000.
H. Garcia-Molina et al., Database Systems: The Complete Book. Pearson, 2019.
S. Abiteboul, R. Hull, and V. Vianu, Foundations of Databases. Addison-Wesley, 1995.
L. K. McDowell and J. A. Hendler, “Semantic web and AI integration,” IEEE Intell. Syst., vol. 27, no. 6, pp. 86–90, 2012.
H. Jagadish et al., “Big data and knowledge extraction,” Commun. ACM, vol. 59, no. 11, pp. 86–96, 2016.
P. M. Dorfman, “Automated data profiling systems,” U.S. Patent 9 121 998, Sep. 1, 2015.
S. K. Lakshmanan et al., “Automated data transformation with meta-learning,” Proc. SIGMOD, pp. 131–147, 2022.
Y. Sun, “Self-supervised learning for tabular data,” arXiv:2110.01839, 2021.
AWS, “AWS Glue: A fully managed ETL service,” aws.amazon.com, 2022.
Databricks, “Unity Catalog: Fine-grained governance for Lakehouse,” databricks.com, 2022.
Google, “Dataflow automation,” cloud.google.com, 2022.
Microsoft, “Fabric Lakehouse architecture,” microsoft.com, 2023.
A. Halevy et al., “The unfolding human and machine intelligence for data integration,” Proc. VLDB, 2020.
P. Domingos, “A few useful things to know about machine learning,” Commun. ACM, vol. 55, no. 10, pp. 78–87, 2012.
A. Ng, “Machine learning yearning,” deeplearning.ai, 2018.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv:1412.6980, 2015.
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Comput., vol. 9, pp. 1735–1780, 1997.
A. Vaswani et al., “Attention is all you need,” in Adv. Neural Inf. Process. Syst., 2017.
F. Chollet, Deep Learning with Python. Manning, 2018.
J. Yang et al., “AI-planning for autonomous data pipelines,” in Proc. AAAI, 2021.
L. Li et al., “AutoML for data engineering tasks,” IEEE Trans. Knowl. Data Eng., vol. 32, no. 8, pp. 1537–1548, 2020.
M. Bernecker and H. Plattner, “Self-healing data pipelines,” Proc. ICDE, pp. 201–212, 2020.
IBM, “Watson AIOps: Automating data-intensive operations,” 2021.
J. S. Anderson, “Full-stack AI data engineering systems,” ACM Queue, vol. 19, no. 4, pp. 45–72, 2021.
J. L. Hellerstein et al., “The MADlib analytics library,” Proc. VLDB, vol. 5, no. 12, pp. 1700–1711, 2012.
A. Rajaraman and J. Ullman, Mining of Massive Datasets. Cambridge Univ. Press, 2014.
M. Chen et al., “Evaluating LLMs for structured data tasks,” arXiv:2308.01234, 2023.
O. Press, “Emergent abilities of large language models,” Commun. ACM, 2024.
NVIDIA, “AI agents for autonomous workflows,” developer.nvidia.com, 2023.
M. T. Ribeiro, S. Singh, and C. Guestrin, “Why should I trust you? Explaining ML predictions,” in Proc. KDD, 2016.
J. Manyika et al., “The future of artificial intelligence,” McKinsey Global Institute, 2023.
S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach, 4th ed., Pearson, 2020.
D. Jurafsky and J. H. Martin, Speech and Language Processing, 3rd ed., Draft, 2023.
T. Dietterich, “Steps toward robust artificial intelligence,” AI Mag., vol. 38, no. 3, pp. 3–24, 2017.
R. Shwartz-Ziv and A. Armon, “Tabular data: Deep learning is not all you need,” arXiv:2106.03253, 2021.
Z. Zhang et al., “A survey on reinforcement learning,” IEEE Trans. Neural Netw. Learn. Syst., 2021.
J. Schulman et al., “Proximal policy optimization algorithms,” arXiv:1707.06347, 2017.
O. Vinyals et al., “Grandmaster level in StarCraft II using multi-agent reinforcement learning,” Nature, 2019.
B. Settles, Active Learning, Morgan & Claypool, 2012.
T. Chen et al., Introduction to Machine Learning Using Python, O’Reilly, 2016.
H. Larochelle et al., “Learning algorithms for deep architectures,” Found. Trends Mach. Learn., 2009.
L. Bottou, “Stochastic gradient descent tricks,” in Neural Networks: Tricks of the Trade, 2012.
A. Rahimi and B. Recht, “Random features for large-scale kernel machines,” in Proc. NIPS, 2007.
G. Hinton et al., “Improving neural networks by preventing co-adaptation,” arXiv:1207.0580, 2012.
N. Srivastava et al., “Dropout: A simple way to prevent overfitting,” JMLR, 2014.
Y. Gal and Z. Ghahramani, “Dropout as Bayesian approximation,” Proc. ICML, 2016.
D. Dua and C. Graff, “UCI machine learning repository,” 2017.
J. Kahn et al., “Self-supervised learning for sequential data,” arXiv:2010.11647, 2020.
A. Graves, “Generating sequences with RNNs,” arXiv:1308.0850, 2013.
Y. Bengio et al., “Curriculum learning,” Proc. ICML, 2009.
A. G. Baydin et al., “Automatic differentiation in ML,” JMLR, 2018.
B. Zhou et al., “Interpretable deep learning,” arXiv:1812.06499, 2019.
M. T. Ribeiro et al., “Anchors: High-precision model-agnostic explanations,” Proc. AAAI, 2018.
Google, “Vertex AI: Unified ML platform,” cloud.google.com, 2023.
Meta AI, “LLaMA: Open and efficient foundation models,” arXiv:2302.13971, 2023.
Anthropic, “Constitutional AI: Harmlessness from AI principles,” arXiv:2212.08073, 2022.
A. Radford et al., “Language models are unsupervised multitask learners,” OpenAI Technical Report, 2019.
J. Zeng et al., “A survey on foundation models,” arXiv:2302.05284, 2023.
H. Zhang et al., “Mixup: Beyond empirical risk minimization,” Proc. ICLR, 2018.
A. Krizhevsky et al., “ImageNet classification with deep convolutional networks,” Commun. ACM, 2017.
P. J. Rousseeuw, “Silhouettes: A graphical aid to cluster validation,” J. Comput. Appl. Math., 1987.
M. Jordan and T. Mitchell, “Machine learning: Trends, perspectives, and prospects,” Science, 2015.
H. Choi et al., “Evaluation of LLMs for real-world decision-making,” arXiv:2309.06275, 2023.
J. Pearl, Causality: Models, Reasoning and Inference, Cambridge Univ. Press, 2009.
G. Roelofs et al., “Responsible AI: Best practices,” Google Research, 2022.
Microsoft, “Responsible AI Standard,” 2022.
NIST, “AI Risk Management Framework,” U.S. Department of Commerce, 2023.
G. Marcus and E. Davis, Rebooting AI, Pantheon Books, 2019.
E. Brynjolfsson and A. McAfee, The Second Machine Age, Norton, 2014.
D. Sculley et al., “Hidden technical debt in ML systems,” Proc. NIPS, 2015.
R. Mayer et al., “Data lineage in modern data systems,” Proc. VLDB, 2021.
V. Kumar et al., “Survey on anomaly detection in streaming data,” ACM Comput. Surv., 2022.
C. Aggarwal, Data Streams: Models and Algorithms, Springer, 2007.
L. Bonomi et al., “Knowledge graphs: Foundations and applications,” Proc. IEEE, 2022.
V. Lopez et al., “Ontology-based data analysis,” Semantic Web J., 2021.
P. Alipanahi et al., “Predicting the sequence specificities of DNA-binding proteins,” Nature, 2015.
T. Salimans et al., “Evolution strategies as scalable alternatives to RL,” arXiv:1703.03864, 2017.
Salesforce, “AI Agents for enterprise automation,” salesforce.com, 2023.
Snowflake, “Dynamic query optimization for modern data platforms,” snowflake.com, 2022.
Gartner, “Hype Cycle for Artificial Intelligence,” Gartner Research, 2023.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Koteswara Rao Chirumamilla
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.
IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
