Explainability Measurement of Machine Learning Model in Phishing Detection
Keywords:
Phishing Detection, Machine Learning, Explainability Metric, URL, FeaturesAbstract
Explainability in phishing detection models can enhance phishing assault mitigation by fostering confidence and elucidating the detection process. The essential requirements for facilitating human comprehension and assessment of the reasons a specific URL is deemed insecure for visitation. The aims of this study are to investigate some machine learning models in phishing detection which have abilities to fulfil the critical needs of explanation using explainability metric. This study applies a methodology starting with dataset collection of phishing and legitimate URL as the sources of various features. Then the models selected, which are often known have good quality in classification between phishing or legitimate label. The modeling results are processed using an explainer method to generate a comprehensive understanding of feature behaviors that influence model predictions. Instead of present accuracy metric results only, this study discusses how explainability metric shows how the features contribute to the model. The conclusion shows that some features have abilities to influence the model decision in general or specifically, then how the features contribute to the model in terms of stability and distribution behaviors. The study shows that some features that may be identified as key features of model behavior then can be applied practically to phishing detection systems such as firewall or SIEM (Security Information and Event Management).
Downloads
References
M. Das, S. Saraswathi, R. Panda, A. Mishra, and ..., “Exquisite analysis of popular machine learning–based phishing detection techniques for cyber systems,” Journal of Applied …, no. Query date: 2023-03-02 08:19:27, 2021, doi: 10.1080/19361610.2020.1816440.
W. Saeed and C. Omlin, “Explainable AI (XAI): A Systematic Meta-Survey of Current Challenges and Future Opportunities,” Knowl. Based Syst., vol. 263, p. 110273, 2021, doi: 10.1016/j.knosys.2023.110273.
A. Nadeem, D. Vos, C. Cao, L. Pajola, and ..., “Sok: Explainable machine learning for computer security applications,” 2023 IEEE 8th …, 2023, [Online]. Available: https://ieeexplore.ieee.org/abstract/document/10190524/
A. Warnecke, D. J. Arp, C. Wressnegger, and K. Rieck, “Don’t Paint It Black: White-Box Explanations for Deep Learning in Computer Security.,” Cornell University, Jun. 2019, [Online]. Available: https://arxiv.org/abs/1906.02108v1
C. Rudin, C. Chen, Z. Chen, H. Huang, L. Semenova, and C. Zhong, “Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges,” Cornell University. Jan. 2021. doi: 10.48550/arXiv.2103.
F. Charmet, T. Morikawa, A. Tanaka, and T. Takahashi, “VORTEX : Visual phishing detectiOns aRe Through EXplanations,” ACM Trans. Internet Technol., vol. 24, no. 2, pp. 1–24, May 2024, doi: 10.1145/3654665.
G. Ramesh, “Identification of phishing webpages and its target domains by analyzing the feign relationship,” Journal of Information Security and Applications, vol. 35, no. Query date: 2024-02-17 23:38:53, pp. 75–84, 2017, doi: 10.1016/j.jisa.2017.06.001.
S. Mittal, “Explaining URL Phishing Detection by Glass Box Models,” Proceedings of the 2023 Fifteenth International Conference on Contemporary Computing, 2023, [Online]. Available: https://api.semanticscholar.org/CorpusID:263147290
M. C. Calzarossa, P. Giudici, and R. Zieni, “Explainable Machine Learning for Bag of Words-Based Phishing Detection,” … on Explainable Artificial Intelligence, 2023, doi: 10.1007/978-3-031-44064-9_28.
M. Affenzeller et al., “White Box vs. Black Box Modeling: On the Performance of Deep Learning, Random Forests, and Symbolic Regression in Solving Regression Problems,” in International Conference/Workshop on Computer Aided Systems Theory, 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID:215791513
O. Loyola-González, “Black-Box vs. White-Box: Understanding Their Advantages and Weaknesses From a Practical Point of View,” IEEE Access, vol. 7, pp. 154096–154113, 2019.
C. Bentéjac, A. Csörgo, and G. Martínez-Muñoz, “A comparative analysis of gradient boosting algorithms,” Artificial Intelligence Review, vol. 54, pp. 1937–1967, 2019.
E. M. Kenny, C. Ford, M. Quinn, and M. T. Keane, “Explaining black-box classifiers using post-hoc explanations-by-example: The effect of explanations and error-rates in XAI user studies,” Artificial Intelligence. Elsevier, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0004370221000102
L. Ostroumova, G. Gusev, A. Vorobev, A. V. Dorogush, and A. Gulin, “CatBoost: unbiased boosting with categorical features,” in Neural Information Processing Systems, 2017. [Online]. Available: https://api.semanticscholar.org/CorpusID:5044218
J. T. Hancock and T. M. Khoshgoftaar, “CatBoost for big data: an interdisciplinary review,” Journal of Big Data, vol. 7, 2020, [Online]. Available: https://api.semanticscholar.org/CorpusID:226254770
A. E. Maxwell, M. Sharma, and K. A. Donaldson, “Explainable Boosting Machines for Slope Failure Spatial Predictive Modeling,” Remote Sensing, vol. 13, no. 24, p. 4991, Dec. 2021.
R. Massafra et al., “Analyzing breast cancer invasive disease event classification through explainable artificial intelligence,” Frontiers Media, vol. 10, Feb. 2023, doi: 10.3389/fmed.2023.1116354.
H. Kaneko, “Interpretation of Machine Learning Models for Data Sets with Many Features Using Feature Importance,” American Chemical Society, vol. 8, no. 25, pp. 23218–23225, Jun. 2023, doi: 10.1021/acsomega.3c03722.
K. Aas, M. Jullum, and A. Løland, “Explaining individual predictions when features are dependent: More accurate approximations to Shapley values,” Elsevier BV, vol. 298, pp. 103502–103502, Mar. 2021, doi: 10.1016/j.artint.2021.103502.
M. Philipp, T. Rusch, K. Hornik, and C. Strobl, “Measuring the Stability of Results From Supervised Statistical Learning,” Taylor & Francis, vol. 27, no. 4, pp. 685–700, May 2018, doi: 10.1080/10618600.2018.1473779.
W. J. Murdoch, C. Singh, K. Kumbier, R. Abbasi-Asl, and B. Yu, “Definitions, methods, and applications in interpretable machine learning,” Proceedings of the National Academy of Sciences, vol. 116, no. 44, pp. 22071–22080, 2019, doi: 10.1073/pnas.1900654116.
S. Lundberg and S.-I. Lee, “A Unified Approach to Interpreting Model Predictions,” Cornell University. Jan. 2017. doi: 10.48550/arXiv.1705.
S. R. Sharma, “A Feature Selection Comparative Study for Web Phishing Datasets,” Proceedings of CONECCT 2020 - 6th IEEE International Conference on Electronics, Computing and Communication Technologies, no. Query date: 2024-02-17 23:37:56, 2020, doi: 10.1109/CONECCT50063.2020.9198349.
I. Covert, S. Lundberg, and S.-I. Lee, “Understanding Global Feature Contributions With Additive Importance Measures,” Cornell University. Jan. 2020. doi: 10.48550/arXiv.2004.
H. Faris and S. Yazid, “Phishing Web Page Detection Methods: URL and HTML Features Detection,” … IEEE International Conference on Internet of …, no. Query date: 2023-03-02 08:19:27, 2021, [Online]. Available: https://ieeexplore.ieee.org/abstract/document/9359694/
L. F. Gutiérrez and A. S. Namin, “Generating Interpretable Features for Context-Aware Document Clustering: A Cybersecurity Case Study,” … Conference on Big Data (Big Data), 2022, [Online]. Available: https://ieeexplore.ieee.org/abstract/document/10021049/
L. H. Gilpin, D. Bau, B. Z. Yuan, A. Bajwa, M. Specter, and L. Kagal, “Explaining Explanations: An Overview of Interpretability of Machine Learning,” Cornell University. Jan. 2018. doi: 10.48550/arxiv.1806.00069.
P. Maneriker, J. Stokes, E. Lazo, and ..., “URLTran: Improving phishing URL detection using transformers,” MILCOM 2021-2021 …, no. Query date: 2023-03-02 08:19:27, 2021, [Online]. Available: https://ieeexplore.ieee.org/abstract/document/9653028/
S. Dalvi, G. Gressel, and K. Achuthan, “Tuning the false positive rate/false negative rate with phishing detection models,” Int. J. Eng. Adv. Technol, no. Query date: 2023-03-02 08:19:27, 2019, [
F. S. Bidabadi and S. Wang, “A new weighted ensemble model for phishing detection based on feature selection,” Cornell University. Jan. 2022. doi: 10.48550/arxiv.2212.11125.
H. Yuan, “Detecting Phishing Websites and Targets Based on URLs and Webpage Links,” Proceedings - International Conference on Pattern Recognition, vol. 2018, no. Query date: 2024-02-17 23:38:53, pp. 3669–3674, 2018, doi: 10.1109/ICPR.2018.8546262.
H. Zuhair, “New hybrid features for phish website prediction,” International Journal of Advances in Soft Computing and its Applications, vol. 8, no. 1, pp. 28–43, 2016.
A. Basit, M. Zafar, X. Liu, A. R. Javed, Z. Jalil, and K. Kifayat, “A comprehensive survey of AI-enabled phishing attacks detection techni ques,” Telecommun Syst, vol. 76, no. 1, pp. 139–154, Oct. 2020, doi: 10.1007/s11235-020-00733-2.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.
IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
