Mitigating Data Drift in Distributed Streaming Pipelines for Real-Time Financial Fraud Detection
Keywords:
deployments, detection, Population, sensitivityAbstract
Fraud detection systems built on real-time data streams must contend with a persistent challenge: transaction data does not stay stationary. As attackers adapt, merchant patterns shift, and economic conditions change, the statistical profile of both legitimate and fraudulent activity evolves continuously—a phenomenon called concept drift that erodes classifier accuracy and, absent remediation, renders deployed models unreliable within weeks. The challenge compounds in distributed streaming environments, where high-velocity data arrives across heterogeneous partitions and centralized monitoring becomes computationally impractical. This paper introduces a multi-layer drift detection and adaptation framework tailored to distributed financial fraud pipelines. Three detection mechanisms work in tandem: Adaptive Windowing (ADWIN) handles memory-efficient stream segmentation, the Drift Detection Method (DDM) tracks error rates against statistical control thresholds, and the Population Stability Index (PSI) monitors individual feature distributions. Each mechanism targets a different temporal scale and sensitivity level, so the system can raise early warnings before downstream classification quality visibly degrades. A selective ensemble retraining strategy activates on confirmed drift events while leaving stable feature subsets undisturbed. Experiments on synthetic and publicly available financial stream benchmarks confirm that the layered approach shortens detection delay and lowers false alarm rates compared to single-detector configurations. The framework targets Apache Kafka and Apache Flink deployments; operator-level computational overhead is characterized throughout.
Downloads
References
Bayram, F., Ahmed, B. S., & Kassler, A. (2022). From concept drift to model degradation: An overview on performance-aware drift detectors. ACM Computing Surveys, 55(4), 1–43. https://doi.org/10.1145/3523152
Bifet, A., & Gavaldà, R. (2007). Learning from time-changing data with adaptive windowing. Proceedings of the 2007 SIAM International Conference on Data Mining, 443–448. https://doi.org/10.1137/1.9781611972771.42
Gama, J., Medas, P., Castillo, G., & Rodrigues, P. (2004). Learning with drift detection. In Advances in Artificial Intelligence – SBIA 2004 (pp. 286–295). Springer. https://doi.org/10.1007/978-3-540-28645-5_29
Zliobaite, I. (2010). Learning under concept drift: An overview. arXiv preprint arXiv:1010.4784. https://arxiv.org/abs/1010.4784
Lu, J., Liu, A., Dong, F., Gu, F., Gama, J., & Zhang, G. (2018). Learning under concept drift: A review. IEEE Transactions on Knowledge and Data Engineering, 31(12), 2346–2363. https://doi.org/10.1109/TKDE.2018.2876857
Gomes, H. M., Bifet, A., Read, J., Barddal, J. P., Enembreck, F., Pfharinger, B., Holmes, G., & Abdessalem, T. (2017). Adaptive random forests for evolving data stream classification. Machine Learning, 106(9), 1469–1495. https://doi.org/10.1007/s10994-017-5642-8
Cassales, G. W., Gomes, H. M., Bifet, A., Pfahringer, B., & Senger, H. (2019). Improving the performance of bagging ensembles for data streams through parallelism. Proceedings of the International Joint Conference on Neural Networks (IJCNN), 1–8. https://doi.org/10.1109/IJCNN.2019.8852241
Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F., & Dennison, D. (2015). Hidden technical debt in machine learning systems. Advances in Neural Information Processing Systems, 28, 2503–2511. https://proceedings.neurips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
Ditzler, G., Roveri, M., Alippi, C., & Polikar, R. (2015). Learning in nonstationary environments: A survey. IEEE Computational Intelligence Magazine, 10(4), 12–25. https://doi.org/10.1109/MCI.2015.2471196
Wang, H., Fan, W., Yu, P. S., & Han, J. (2003). Mining concept-drifting data streams using ensemble classifiers. Proceedings of the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 226–235. https://doi.org/10.1145/956750.956778
Assis, L. P., & Souza, V. M. A. (2025). ADWIN-U: An unsupervised adaptive windowing method for streaming data. Expert Systems with Applications, 263, 125780. https://doi.org/10.1016/j.eswa.2024.125780
Webb, G. I., Hyde, R., Cao, H., Nguyen, H. L., & Petitjean, F. (2016). Characterizing concept drift. Data Mining and Knowledge Discovery, 30(4), 964–994. https://doi.org/10.1007/s10618-015-0448-4
Gomes, H. M., Read, J., Bifet, A., Barddal, J. P., & Gama, J. (2019). Machine learning for streaming data: State of the art, challenges, and opportunities. ACM SIGKDD Explorations Newsletter, 21(2), 6–22. https://doi.org/10.1145/3373464.3373470
Street, W. N., & Kim, Y. (2001). A streaming ensemble algorithm (SEA) for large-scale classification. Proceedings of the 7th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 377–382. https://doi.org/10.1145/502512.502568
Bifet, A., Holmes, G., Kirkby, R., & Pfahringer, B. (2010). MOA: Massive online analysis. Journal of Machine Learning Research, 11, 1601–1604. https://jmlr.org/papers/v11/bifet10a.html
Gonçalves, P. M., de Carvalho Santos, S. G. T., Barros, R. S. M., & Vieira, D. C. L. (2014). A comparative study on concept drift detectors. Expert Systems with Applications, 41(18), 8144–8156. https://doi.org/10.1016/j.eswa.2014.07.017
Souza, V. M. A., dos Reis, D. M., Maletzke, A. G., & Batista, G. E. A. P. A. (2020). Challenges in benchmarking stream learning algorithms with real-world data. Data Mining and Knowledge Discovery, 34(6), 1805–1858. https://doi.org/10.1007/s10618-020-00698-5
Brzezinski, D., & Stefanowski, J. (2014). Reacting to different types of concept drift: The accuracy updated ensemble algorithm. IEEE Transactions on Neural Networks and Learning Systems, 25(1), 81–94. https://doi.org/10.1109/TNNLS.2013.2251352
Gomes, H. M., Bifet, A., & Pfahringer, B. (2022). Semi-supervised learning over streaming data. Machine Learning, 111(11), 3971–4010. https://doi.org/10.1007/s10994-022-06186-9
Krawczyk, B., Minku, L. L., Gama, J., Stefanowski, J., & Wozniak, M. (2017). Ensemble learning for data stream analysis: A survey. Information Fusion, 37, 132–156. https://doi.org/10.1016/j.inffus.2017.02.004
Losing, V., Hammer, B., & Wersing, H. (2018). Incremental on-line learning: A review and comparison of state of the art algorithms. Neurocomputing, 275, 1261–1274. https://doi.org/10.1016/j.neucom.2017.06.084
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.
IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
