Efficiency-Enhanced Densenet Architectures: An Exploration of Multi-Kernel, Multi-Branch Structures for Achieving Optimal Trade-Off Between Parameters and Accuracy
Keywords:
One-Layer Structure, DenseNet, Dense-Block, Optimized Network, Convolutional Neural NetworkAbstract
In this work, we present intricately woven investigative one-layer structure designs for the Dense-Block of the DenseNet, intending to improve performance in visual recognition tasks. Developing a robust representation for accurate visual recognition is a critical challenge that requires more than just increasing the depth and width of neural networks. Therefore, we have devoted significant effort to developing new One-Layer Structures (OLSs) for the Dense-Block. Our proposed OLSs are comprising multiple branches of stacks of 1×1, 3×3, and 5×5 convolutional layers. We recommend replacing the standard OLS of Dense-Block with one of these proposed OLSs. Our proposed OLSs are lightweight, simple, and optimally arranged, making them an ideal choice for optimizing network performance. We organized them into three families: 1.0 and 1.1, 2.0 to 2.3, and 3.0 to 3.3. To evaluate the effectiveness of our proposed models, we conduct experiments on the three benchmark datasets: Imagenette, CIFAR-10, and CIFAR-100. The investigation of DenseNet models enhanced with OLSs up to version 3.3 provides a nuanced understanding of the intricate relationship between model complexity, computational efficiency, and accuracy. Through meticulous analysis on multiple datasets, a consistent pattern of parameter and FLOP reduction is observed, indicating progressive refinement in model architecture. OLS 2.X versions achieve accuracy values ranging from 94.84% to 95.31% on CIFAR-10, 80.33% to 80.69% on CIFAR-100, and 93.325% to 93.478% on Imagenette demonstrating that the integration of OLSs contributes positively to model performance.
Downloads
References
Liu S, Deng W. Very deep convolutional neural network based image classification using small training sample size. In2015 3rd IAPR Asian conference on pattern recognition (ACPR) 2015 Nov 3 (pp. 730-734). IEEE.
Nair V, Hinton GE. Rectified linear units improve restricted boltzmann machines. InProceedings of the 27th international conference on machine learning (ICML-10) 2010 (pp. 807-814).
Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A. Going deeper with convolutions. InProceedings of the IEEE conference on computer vision and pattern recognition 2015 (pp. 1-9).
Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z. Rethinking the inception architecture for computer vision. InProceedings of the IEEE conference on computer vision and pattern recognition 2016 (pp. 2818-2826).
Szegedy C, Ioffe S, Vanhoucke V, Alemi A. Inception-v4, inception-resnet and the impact of residual connections on learning. InProceedings of the AAAI conference on artificial intelligence 2017 Feb 12 (Vol. 31, No. 1).
He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition 2016 (pp. 770-778).
Huang G, Liu Z, Van Der Maaten L, Weinberger KQ. Densely connected convolutional networks. InProceedings of the IEEE conference on computer vision and pattern recognition 2017 (pp. 4700-4708).
Girshick R. Fast r-cnn. InProceedings of the IEEE international conference on computer vision 2015 (pp. 1440-1448).
Girshick R, Donahue J, Darrell T, Malik J. Rich feature hierarchies for accurate object detection and semantic segmentation. InProceedings of the IEEE conference on computer vision and pattern recognition 2014 (pp. 580-587).
Bendou Y, Hu Y, Lafargue R, Lioi G, Pasdeloup B, Pateux S, Gripon V. Easy—ensemble augmented-shot-y-shaped learning: State-of-the-art few-shot classification with simple components. Journal of Imaging. 2022 Jun 24;8(7):179.
Ridnik T, Sharir G, Ben-Cohen A, Ben-Baruch E, Noy A. Ml-decoder: Scalable and versatile classification head. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision 2023 (pp. 32-41).
Jin C, Liang J, Fan C, Chen L, Wang Q, Lu Y, Wang K. Study on segmentation of blasting fragment images from open-pit mine based on U-CARFnet. Plos one. 2023 Sep 14;18(9):e0291115.
Shen X, Wang H, Wei B, Cao J. Real-time scene classification of unmanned aerial vehicles remote sensing image based on Modified GhostNet. Plos one. 2023 Jun 7;18(6):e0286873.
Hassani A, Walton S, Shah N, Abuduweili A, Li J, Shi H. Escaping the big data paradigm with compact transformers. arXiv preprint arXiv:2104.05704. 2021 Apr 12.
Savarese P, Maire M. Learning implicitly recurrent CNNs through parameter sharing. arXiv preprint arXiv:1902.09701. 2019 Feb 26.
Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861. 2017 Apr 17.
Chiang HY, Frumkin N, Liang F, Marculescu D. MobileTL: on-device transfer learning with inverted residual blocks. InProceedings of the AAAI Conference on Artificial Intelligence 2023 Jun 26 (Vol. 37, No. 6, pp. 7166-7174).
Zhou D, Hou Q, Chen Y, Feng J, Yan S. Rethinking bottleneck structure for efficient mobile network design. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16 2020 (pp. 680-697). Springer International Publishing.
Han K, Wang Y, Tian Q, Guo J, Xu C, Xu C. Ghostnet: More features from cheap operations. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition 2020 (pp. 1580-1589).
Zhang X, Zhou X, Lin M, Sun J. Shufflenet: An extremely efficient convolutional neural network for mobile devices. InProceedings of the IEEE conference on computer vision and pattern recognition 2018 (pp. 6848-6856).
Zhang H, Zhu X, Li B, Guan Z, Che W. LA-ShuffleNet: A Strong Convolutional Neural Network for Edge Computing Devices. IEEE Access. 2023 Oct 16.
Huang G, Liu S, Van der Maaten L, Weinberger KQ. Condensenet: An efficient densenet using learned group convolutions. InProceedings of the IEEE conference on computer vision and pattern recognition 2018 (pp. 2752-2761).
Li X, Wang W, Hu X, Yang J. Selective kernel networks. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition 2019 (pp. 510-519).
Wang F, Jiang M, Qian C, Yang S, Li C, Zhang H, Wang X, Tang X. Residual attention network for image classification. InProceedings of the IEEE conference on computer vision and pattern recognition 2017 (pp. 3156-3164).
Fu J, Liu J, Tian H, Li Y, Bao Y, Fang Z, Lu H. Dual attention network for scene segmentation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition 2019 (pp. 3146-3154).
Chen L, Zhang H, Xiao J, Nie L, Shao J, Liu W, Chua TS. Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning. InProceedings of the IEEE conference on computer vision and pattern recognition 2017 (pp. 5659-5667).
Chen S, Liu Y, Gao X, Han Z. Mobilefacenets: Efficient cnns for accurate real-time face verification on mobile devices. InBiometric Recognition: 13th Chinese Conference, CCBR 2018, Urumqi, China, August 11-12, 2018, Proceedings 13 2018 (pp. 428-438). Springer International Publishing.
Li H, Xiong P, An J, Wang L. Pyramid attention network for semantic segmentation. arXiv preprint arXiv:1805.10180. 2018 May 25.
Jiang Y, Cheng T, Dong J, Liang J, Zhang Y, Lin X, Yao H. Dermoscopic image segmentation based on Pyramid Residual Attention Module. Plos one. 2022 Sep 16;17(9):e0267380.
Choi M, Kim H, Han B, Xu N, Lee KM. Channel attention is all you need for video frame interpolation. InProceedings of the AAAI Conference on Artificial Intelligence 2020 Apr 3 (Vol. 34, No. 07, pp. 10663-10671).
Ioffe S, Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning 2015 Jun 1 (pp. 448-456). pmlr.
Hu J, Shen L, Sun G. Squeeze-and-excitation networks. InProceedings of the IEEE conference on computer vision and pattern recognition 2018 (pp. 7132-7141).
Zhang T, Qi GJ, Xiao B, Wang J. Interleaved group convolutions. InProceedings of the IEEE international conference on computer vision 2017 (pp. 4373-4382).
Ullrich K, Meeds E, Welling M. Soft weight-sharing for neural network compression. arXiv preprint arXiv:1702.04008. 2017 Feb 13.
Han S, Mao H, Dally WJ. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149. 2015 Oct 1.
Zhang X, Colbert I, Kreutz-Delgado K, Das S. Training deep neural networks with joint quantization and pruning of weights and activations. arXiv preprint arXiv:2110.08271. 2021 Oct 15.
Lu Q, Jiang W, Xu X, Hu J, Shi Y. Quantization Through Search: A Novel Scheme to Quantize Convolutional Neural Networks in Finite Weight Space. InProceedings of the 28th Asia and South Pacific Design Automation Conference 2023 Jan 16 (pp. 378-383).
Choi Y, El-Khamy M, Lee J. Towards the limit of network quantization. arXiv preprint arXiv:1612.01543. 2016 Dec 5.
Pham H, Guan M, Zoph B, Le Q, Dean J. Efficient neural architecture search via parameters sharing. In International conference on machine learning 2018 Jul 3 (pp. 4095-4104). PMLR.
Liu C, Zoph B, Neumann M, Shlens J, Hua W, Li LJ, Fei-Fei L, Yuille A, Huang J, Murphy K. Progressive neural architecture search. InProceedings of the European conference on computer vision (ECCV) 2018 (pp. 19-34).
Real E, Aggarwal A, Huang Y, Le QV. Regularized evolution for image classifier architecture search. InProceedings of the aaai conference on artificial intelligence 2019 Jul 17 (Vol. 33, No. 01, pp. 4780-4789).
Guo X, Wu Y, Miao J, Chen Y. LiteGaze: Neural architecture search for efficient gaze estimation. Plos one. 2023 May 1;18(5):e0284814.
Zoph B, Vasudevan V, Shlens J, Le QV. Learning transferable architectures for scalable image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition 2018 (pp. 8697-8710).
Radosavovic I, Kosaraju RP, Girshick R, He K, Doll´ar P. Designing network design spaces. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition 2020 (pp. 10428-10436).
Ding X, Zhang X, Han J, Ding G. Diverse branch block: Building a convolution as an inception-like unit. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2021 (pp. 10886-10895).
Ding X, Guo Y, Ding G, Han J. Acnet: Strengthening the kernel skeletons for powerful cnn via asymmetric convolution blocks. InProceedings of the IEEE/CVF international conference on computer vision 2019 (pp. 1911-1920).
Ding X, Zhang X, Ma N, Han J, Ding G, Sun J. Repvgg: Making vgg-style convnets great again. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition 2021 (pp. 13733-13742).
Krizhevsky A, Nair V, Hinton G. Cifar-10 (canadian institute for advanced research). URL http://www. cs. toronto. edu/kriz/cifar. html. 2010 Mar;5(4):1.
Howard J. Imagewang. URL https://github. com/fastai/imagenette. 2019.
He K, Zhang X, Ren S, Sun J. Identity mappings in deep residual networks. InComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14 2016 (pp. 630-645). Springer International Publishing.
Gross S, Wilber M. Training and investigating residual nets. Facebook AI Research. 2016 May;6(3).
He K, Zhang X, Ren S, Sun J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. InProceedings of the IEEE international conference on computer vision 2015 (pp. 1026-1034).
Mormille LH, Broni-Bediako C, Atsumi M. Regularizing self-attention on vision transformers with 2D spatial distance loss. Artificial Life and Robotics. 2022 Aug;27(3):586-93.
Remerscheid NW, Ziller A, Rueckert D, Kaissis G. Smoothnets: Optimizing cnn architecture design for differentially private deep learning. arXiv preprint arXiv:2205.04095. 2022 May 9.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.
IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
