Neural Network Pruning Techniques for Efficient Model Compression
Keywords:
Neural Network Pruning, compression, deep learning, performance, accuracyAbstract
A network of neurons When it comes to meeting the growing need for deploying deep learning models on devices with limited resources, pruning has emerged as an essential strategy for model reduction. The purpose of this study is to offer a detailed review of several pruning approaches that attempt to reduce the size and computational complexity of neural networks while maintaining their predictive accuracy. Specifically, the major emphasis is placed on structured pruning techniques, which include the removal of whole neurons, channels, or layers in a methodical manner based on certain criteria. In this article, we go into the fundamental ideas that underlie magnitude-based pruning, weight clustering, and filter pruning, and we emphasize the usefulness of these techniques in achieving considerable model reduction. In addition, this work investigates the interaction between pruning procedures and fine-tuning tactics in order to reduce the possibility of accuracy loss. In addition, the research investigates unstructured pruning techniques, which entail the elimination of individual weights in order to bring about sparsity in the network. The difficulties that are connected with unstructured pruning are discussed, and methods such as iterative pruning and regularization procedures are investigated as potential ways to improve the effectiveness of this kind of pruning. The comparative comparison of these different pruning strategies gives insight on the advantages, disadvantages, and compromises associated with each of them. Additionally, we highlight recent breakthroughs, including the integration of neural architecture search with pruning and the examination of pruning in the context of specialized neural network topologies like transformers.
Downloads
References
X. Zhou, Z. Zhang, L. Wang, and P. Wang, “A Model Based on Siamese Neural Network for Online Transaction Fraud Detection,” Proc. Int. Jt. Conf. Neural Networks, vol. 2019-July, 2019, doi: 10.1109/IJCNN.2019.8852295.
H. Jang and J. Lee, “An Empirical Study on Modeling and Prediction of Bitcoin Prices with Bayesian Neural Networks Based on Blockchain Information,” IEEE Access, vol. 6, pp. 5427–5437, 2017, doi: 10.1109/ACCESS.2017.2779181.
F. Casino, T. K. Dasaklis, and C. Patsakis, “A systematic literature review of blockchain-based applications: Current status, classification and open issues,” Telemat. Informatics, vol. 36, no. November 2018, pp. 55–81, 2019, doi: 10.1016/j.tele.2018.11.006.
S. Teng, G. Chen, G. Liu, J. Lv, and F. Cui, “Modal strain energy-based structural damage detection using convolutional neural networks,” Appl. Sci., vol. 9, no. 16, 2019, doi: 10.3390/app9163376.
S. Liu, A. Borovykh, L. A. Grzelak, and C. W. Oosterlee, “A neural network-based framework for financial model calibration,” J. Math. Ind., vol. 9, no. 1, 2019, doi: 10.1186/s13362-019-0066-7.
S. Tanwar, Q. Bhatia, P. Patel, A. Kumari, P. K. Singh, and W. C. Hong, “Machine Learning Adoption in Blockchain-Based Smart Applications: The Challenges, and a Way Forward,” IEEE Access, vol. 8, pp. 474–448, 2020, doi: 10.1109/ACCESS.2019.2961372.
W. Gao and C. Su, “Analysis on block chain financial transaction under artificial neural network of deep learning,” J. Comput. Appl. Math., vol. 380, p. 112991, 2020, doi: 10.1016/j.cam.2020.112991.
W. Zhang, K. X. Tao, J. F. Li, Y. C. Zhu, and J. Li, “Modeling and Prediction of Stock Price with Convolutional Neural Network Based on Blockchain Interactive Information,” Wirel. Commun. Mob. Comput., vol. 2020, 2020, doi: 10.1155/2020/6686181.
D. Prashar et al., “Blockchain-Based Automated System for Identification and Storage of Networks,” Secur. Commun. Networks, vol. 2021, 2021, doi: 10.1155/2021/6694281.
Z. Zhang, X. Song, L. Liu, J. Yin, Y. Wang, and D. Lan, “Recent Advances in Blockchain and Artificial Intelligence Integration: Feasibility Analysis, Research Issues, Applications, Challenges, and Future Work,” Secur. Commun. Networks, vol. 2021, 2021, doi: 10.1155/2021/9991535.
R. C. Aguilera, M. P. Ortiz, A. A. Banda, and L. E. C. Aguilera, “Blockchain cnn deep learning expert system for healthcare emergency,” Fractals, vol. 29, no. 6, pp. 1–10, 2021, doi: 10.1142/S0218348X21502273.
M. Shafay, R. W. Ahmad, K. Salah, I. Yaqoob, R. Jayaraman, and M. Omar, “Blockchain for Deep Learning : Review and Open Challenges,” no. October, pp. 1–32, 2021, doi: 10.36227/techrxiv.16823140.v1.
T. Frikha, F. Chaabane, N. Aouinti, O. Cheikhrouhou, N. Ben Amor, and A. Kerrouche, “Implementation of Blockchain Consensus Algorithm on Embedded Architecture,” Secur. Commun. Networks, vol. 2021, no. June, 2021, doi: 10.1155/2021/9918697.
F. Jamil, N. Iqbal, Imran, S. Ahmad, and D. Kim, “Peer-to-Peer Energy Trading Mechanism Based on Blockchain and Machine Learning for Sustainable Electrical Power Supply in Smart Grid,” IEEE Access, vol. 9, pp. 39193–39217, 2021, doi: 10.1109/ACCESS.2021.3060457.
Kaushik Dushyant; Garg Muskan; Annu; Ankur Gupta; Sabyasachi Pramanik, "Utilizing Machine Learning and Deep Learning in Cybesecurity: An Innovative Approach," in Cyber Security and Digital Forensics: Challenges and Future Trends , Wiley, 2022, pp.271-293, doi: 10.1002/9781119795667.ch12.
M. Zhu and S. Gupta, “To prune, or not to prune: exploring the efficacy of pruning for model compression.” arXiv, 2017. doi: 10.48550/ARXIV.1710.01878.
V. Talukdar, D. Dhabliya, B. Kumar, S. B. Talukdar, S. Ahamad and A. Gupta, "Suspicious Activity Detection and Classification in IoT Environment Using Machine Learning Approach," 2022 Seventh International Conference on Parallel, Distributed and Grid Computing (PDGC), Solan, Himachal Pradesh, India, 2022, pp. 531-535, doi: 10.1109/PDGC56933.2022.10053312.
Y. Cheng, D. Wang, P. Zhou, and T. Zhang, “A Survey of Model Compression and Acceleration for Deep Neural Networks.” arXiv, 2017. doi: 10.48550/ARXIV.1710.09282.
V. Jain, S. M. Beram, V. Talukdar, T. Patil, D. Dhabliya and A. Gupta, "Accuracy Enhancement in Machine Learning During Blockchain Based Transaction Classification," 2022 Seventh International Conference on Parallel, Distributed and Grid Computing (PDGC), Solan, Himachal Pradesh, India, 2022, pp. 536-540, doi: 10.1109/PDGC56933.2022.10053213.
X. Ruan et al., "EDP: An Efficient Decomposition and Pruning Scheme for Convolutional Neural Network Compression," in IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 10, pp. 4499-4513, Oct. 2021, doi: 10.1109/TNNLS.2020.3018177.
A. Gupta, D. Kaushik, M. Garg and A. Verma, "Machine Learning model for Breast Cancer Prediction," 2020 Fourth International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC), Palladam, India, 2020, pp. 472-477, doi: 10.1109/I-SMAC49090.2020.9243323
Y. Hu, S. Sun, J. Li, X. Wang, and Q. Gu, “A novel channel pruning method for deep neural network compression.” arXiv, 2018. doi: 10.48550/ARXIV.1805.11394.
V. Veeraiah, K. R. Kumar, P. Lalitha Kumari, S. Ahamad, R. Bansal and A. Gupta, "Application of Biometric System to Enhance the Security in Virtual World," 2022 2nd International Conference on Advance Computing and Innovative Technologies in Engineering (ICACITE), Greater Noida, India, 2022, pp. 719-723, doi: 10.1109/ICACITE53722.2022.9823850.
F. Tung and G. Mori, "Deep Neural Network Compression by In-Parallel Pruning-Quantization," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 3, pp. 568-579, 1 March 2020, doi: 10.1109/TPAMI.2018.2886192.
V. Veeraiah, G. P, S. Ahamad, S. B. Talukdar, A. Gupta and V. Talukdar, "Enhancement of Meta Verse Capabilities by IoT Integration," 2022 2nd International Conference on Advance Computing and Innovative Technologies in Engineering (ICACITE), Greater Noida, India, 2022, pp. 1493-1498, doi: 10.1109/ICACITE53722.2022.9823766.
S. Han, H. Mao, and W. J. Dally, “Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding.” arXiv, 2015. doi: 10.48550/ARXIV.1510.00149.
V. Veeraiah, H. Khan, A. Kumar, S. Ahamad, A. Mahajan and A. Gupta, "Integration of PSO and Deep Learning for Trend Analysis of Meta-Verse," 2022 2nd International Conference on Advance Computing and Innovative Technologies in Engineering (ICACITE), Greater Noida, India, 2022, pp. 713-718, doi: 10.1109/ICACITE53722.2022.9823883.
A. Salama, O. Ostapenko, T. Klein, and M. Nabi, “Pruning at a Glance: Global Neural Pruning for Model Compression.” arXiv, 2019. doi: 10.48550/ARXIV.1912.00200.
P. R. Kshirsagar, D. H. Reddy, M. Dhingra, D. Dhabliya and A. Gupta, "A Review on Comparative study of 4G, 5G and 6G Networks," 2022 5th International Conference on Contemporary Computing and Informatics (IC3I), Uttar Pradesh, India, 2022, pp. 1830-1833, doi: 10.1109/IC3I56241.2022.10073385.
L. Deng, G. Li, S. Han, L. Shi and Y. Xie, "Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey," in Proceedings of the IEEE, vol. 108, no. 4, pp. 485-532, April 2020, doi: 10.1109/JPROC.2020.2976475.
P. Venkateshwari, V. Veeraiah, V. Talukdar, D. N. Gupta, R. Anand and A. Gupta, "Smart City Technical Planning Based on Time Series Forecasting of IOT Data," 2023 International Conference on Sustainable Emerging Innovations in Engineering and Technology (ICSEIET), Ghaziabad, India, 2023, pp. 646-651, doi: 10.1109/ICSEIET58677.2023.10303480.
V. Veeraiah, J. Kotti, V. Jain, T. Sharma, S. Saini and A. Gupta, "Scope of IoT in Emerging Engineering Technology during Online Education," 2023 14th International Conference on Computing Communication and Networking Technologies (ICCCNT), Delhi, India, 2023, pp. 1-6, doi: 10.1109/ICCCNT56998.2023.10308107.
Bijender Bansal; V. Nisha Jenipher; Rituraj Jain; R. Dilip; Makhan Kumbhkar; Sabyasachi Pramanik; Sandip Roy; Ankur Gupta, "Big Data Architecture for Network Security," in Cyber Security and Network Security , Wiley, 2022, pp.233-267, doi: 10.1002/9781119812555.ch11.
K. A. Shukla, S. Almal, A. Gupta, R. Jain, R. Mishra and D. Dhabliya, "DL Based System for On-Board Image Classification in Real Time, Applied to Disaster Mitigation," 2022 Seventh International Conference on Parallel, Distributed and Grid Computing (PDGC), Solan, Himachal Pradesh, India, 2022, pp. 663-668, doi: 10.1109/PDGC56933.2022.10053139.
R. Bansal, A. Gupta, R. Singh and V. K. Nassa, "Role and Impact of Digital Technologies in E-Learning amidst COVID-19 Pandemic," 2021 Fourth International Conference on Computational Intelligence and Communication Technologies (CCICT), Sonepat, India, 2021, pp. 194-202, doi: 10.1109/CCICT53244.2021.00046.
A. Gupta, R. Singh, V. K. Nassa, R. Bansal, P. Sharma and K. Koti, "Investigating Application and Challenges of Big Data Analytics with Clustering," 2021 International Conference on Advancements in Electrical, Electronics, Communication, Computing and Automation (ICAECA), Coimbatore, India, 2021, pp. 1-6, doi: 10.1109/ICAECA52838.2021.9675483.
Mamta, V. Veeraiah, D. N. Gupta, B. S. Kumar, A. Gupta and R. Anand, "Prediction of Health Risk Based on Multi-Level IOT Data Using Decision Trees," 2023 International Conference on Sustainable Emerging Innovations in Engineering and Technology (ICSEIET), Ghaziabad, India, 2023, pp. 652-656, doi: 10.1109/ICSEIET58677.2023.10303560.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.
IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
