Governance-as-Code for Intelligent Enterprise Data Governance in Lakehouse Architectures

Authors

  • Himanshu Agarwal

Keywords:

Governance-As-Code, Lakehouse Architecture, Manual, Compliance

Abstract

Lakehouse systems have become part of the storage of large amounts of data in businesses. A lakehouse is an enhanced form of data lake with the data warehouse structure. This mix forms novel issues of data governance. The pace and scale of the new data platform is beyond the ability of manual processes of governance. In this paper, a framework is suggested, which is known as Governance as Code. It is a governance framework that views governance rules as being executed as a piece of code rather than as documents. The regulations are automatically operating within the lakehouse pipeline. The paper provides the structure of this framework, provides a policy coding method and demonstrates the results. This paper presents the argument that Governance as Code will save time to a large extent, enhance the level of consistency and speed in performing a compliance check in an enterprise data platform.

Downloads

Download data is not yet available.

References

Abraham, R., Schneider, J., & Brocke, J. V. (2019). Data governance: A conceptual framework, structured review, and research agenda. International Journal of Information Management, 49, 424–438. https://doi.org/10.1016/j.ijinfomgt.2019.07.008

Alhassan, I., Sammon, D., & Daly, M. (2018). Data governance activities: a comparison between scientific and practice-oriented literature. Journal of Enterprise Information Management, 31(2), 300–316. https://doi.org/10.1108/jeim-01-2017-0007

Azeroual, O., Schöpfel, J., Ivanovic, D., & Nikiforova, A. (2022). Combining data lake and data wrangling for ensuring data quality in CRIS. Procedia Computer Science, 211, 3–16. https://doi.org/10.1016/j.procs.2022.10.171

Begoli, E., Goethert, I., & Knight, K. (2021). A lakehouse architecture for the management and analysis of heterogeneous data for biomedical research and mega-biobanks. 2021 IEEE International Conference on Big Data (Big Data), 4643–4651. https://doi.org/10.1109/bigdata52589.2021.9671534

Beheshti, A., Benatallah, B., Nouri, R., & Tabebordbar, A. (2018). CoreKG. Proceedings of the VLDB Endowment, 11(12), 1942–1945. https://doi.org/10.14778/3229863.3236230

Daga, E., D’Aquin, M., & Motta, E. (2017). Propagating data policies: A user study. In Proceedings of the Knowledge Capture Conference (K-CAP 2017) (pp. 1–8) https://doi.org/10.1145/3148011.3148022

Derakhshannia, M., Gervet, C., Hajj-Hassan, H., Laurent, A., & Martin, A. (2020). Data Lake Governance: towards a systemic and natural ecosystem analogy. Future Internet, 12(8), 126. https://doi.org/10.3390/fi12080126

Eichler, R., Giebler, C., Gröger, C., Schwarz, H., & Mitschang, B. (2021). Modeling metadata in data lakes—A generic model. Data & Knowledge Engineering, 136, 101931. https://doi.org/10.1016/j.datak.2021.101931

Joshi, K. P., Elluri, L., & Nagar, A. (2020). An integrated knowledge graph to automate cloud data compliance. IEEE Access, 8, 148541–148555. https://doi.org/10.1109/access.2020.3008964

Nadal, S., Jovanovic, P., Bilalli, B., & Romero, O. (2022). Operationalizing and automating Data Governance. Journal of Big Data, 9(1), 117. https://doi.org/10.1186/s40537-022-00673-5

Padget, J. A., & Vasconcelos, W. W. (2018). Fine-Grained access control via Policy-Carrying data. ACM Transactions on Internet Technology, 18(3), 1–24. https://doi.org/10.1145/3133324

Sawadogo, P., & Darmont, J. (2021). On data lake architectures and metadata management. Journal of Intelligent Information Systems, 56(1), 97–120. https://doi.org/10.1007/s10844-020-00608-7

Wieder, P., & Nolte, H. (2022). Toward data lakes as central building blocks for data management and analysis. Frontiers in Big Data, 5, 945720. https://doi.org/10.3389/fdata.2022.945720

Yebenes, J., & Zorrilla, M. (2019). Towards a data governance framework for third generation platforms. Procedia Computer Science, 151, 614–621. https://doi.org/10.1016/j.procs.2019.04.082

Zichichi, M., Ferretti, S., D’Angelo, G., & Rodríguez-Doncel, V. (2022). Data governance through a multi-DLT architecture in view of the GDPR. Cluster Computing, 25(6), 4515–4542. https://doi.org/10.1007/s10586-022-03691-3

Downloads

Published

31.07.2023

How to Cite

Himanshu Agarwal. (2023). Governance-as-Code for Intelligent Enterprise Data Governance in Lakehouse Architectures. International Journal of Intelligent Systems and Applications in Engineering, 11(7s), 790–797. Retrieved from https://ijisae.org/index.php/IJISAE/article/view/8536

Issue

Vol. 11 No. 7s (2023): Advancements in Machine Learning for Computer Science and Decision Support Systems

Section

Research Article

License

Creative Commons License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.

IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.