Efficient Framework for Content-Based Image Retrieval using CNN Classification Scores



AlexNet, CBIR, GoogLeNet, ResNet18, Transfer Learning


Content-Based Image retrieval(CBIR) is a technique to search and retrieve similar images from large multimedia databases and an IR system is regarded as efficient if it can retrieve all the images to meet the user’s needs. There are many advanced machine-learning technologies such as deep neural networks(DNN), convolutional neural networks(CNN), and transfer-learning(TL), which are gaining greater importance in image-related tasks. In this paper an efficient framework for content-based image retrieval system adapting transfer-learning on pre-trained CNNs (ResNet18, GoogLeNet, AlexNet) using query-by-image method is proposed, the method explores classification-score descriptors for IR and employ distance metrics for similarity matching. The framework prescribes transfer-learning for efficient retraining of pre-trained CNNs on small datasets chosen from the Wang database. Thirty-plus experiments are designed for finding optimal values of the hyper-parameters and exploring the suitability of six popular distance metrics namely Euclidean, seuclidean, Cityblock, Cosine, Mahalanobis, and Chebychev. After extensive experimentation, a new efficient framework for CBIR using CNN classification scores is proposed and the new framework of CBIR achieves the image retrieval accuracy of 99.45% on natural scene images of 20 classes of the Wang dataset. The experimentations show that the proposed framework is efficient for content-based image retrieval system.


Download data is not yet available.


Prasad, B. E., Gupta, A., Toong, H.-M. D., & Madnick, S. E. (1987), “A Microcomputer-Based Image Database Management System”, IEEE Transactions on Industrial Electronics, IE-34(1), 83–88. https://doi.org/10.1109/tie.1987.350929

Kato, Toshikazu; Jamberdino, Albert A.; Niblack, Carlton W. (1992), “Image Storage and Retrieval Systems – Database architecture for content-based image retrieval”, SPIE Proceedings [SPIE SPIE/IS&T 1992 Symposium on Electronic Imaging: Science and Technology - San Jose, CA (Sunday 9 February 1992)], 1662, 112–123. doi:10.1117/12.58497

Zachary, J.M.; Iyengar, S.S. (1999), “Content based image retrieval systems”, [IEEE Comput. Soc 1999 IEEE Symposium on Application-Specific Systems and Software Engineering and Technology. ASSET'99 - Richardson, TX, USA (24-27 March 1999)] Proceedings 1999 IEEE Symposium on Application-Specific Systems and Software Engineering and Technology. ASSET'99 (Cat. No.PR00122), 136–143. doi:10.1109/asset.1999.756762

Veltkamp, R. C., & Tanase, M. (2002), “A Survey of Content-Based Image Retrieval Systems”, Content-Based Image and Video Retrieval, 47–101. https://doi.org/10.1007/978-1-4615-0987-5_5

Liu, Y., Zhang, D., Lu, G., & Ma, W.-Y. (2007), “A survey of content-based image retrieval with high-level semantics”, Pattern Recognition, 40(1), 262–282. https://doi.org/10.1016/j.patcog.2006.04.045

Rafiee, G. & Dlay, S.s & Woo, Wai Lok. (2010), “A review of content-based image retrieval”, 2010 7th International Symposium on Communication Systems, Networks and Digital Signal Processing, CSNDSP 2010. 775-779. 10.1109/CSNDSP16145.2010.5580313.

Yasmin, M., Mohsin, S., & Sharif, M. (2014), “Intelligent Image Retrieval Techniques: A Survey”, Journal of Applied Research and Technology, 12(1), 87–103. https://doi.org/10.1016/s1665-6423(14)71609-8

Rodrigues, Josiane & Cristo, Marco & Colonna, Juan. (2020), “ Deep hashing for multi-label image retrieval: a survey”, Artificial Intelligence Review. 53. 10.1007/s10462-020-09820-x.

Dubey, S. R. (2021), “A Decade Survey of Content Based Image Retrieval using Deep Learning”, IEEE Transactions on Circuits and Systems for Video Technology, 1–1. https://doi.org/10.1109/tcsvt.2021.3080920

Varga, Domonkos; Sziranyi, Tamas (2016), “Fast content-based image retrieval using convolutional neural network and hash function”, [IEEE 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC) - Budapest, Hungary (2016.10.9-2016.10.12)] 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 002636–002640. doi:10.1109/SMC.2016.7844637

Alzubi, A., Amira, A., & Ramzan, N. (2017), “Content-based image retrieval with compact deep convolutional features”, Neurocomputing, 249, 95–105. https://doi.org/10.1016/j.neucom.2017.03.072

Tzelepi, M., & Tefas, A. (2018), “Deep convolutional learning for Content Based Image Retrieval”, Neurocomputing, 275, 2467–2478. https://doi.org/10.1016/j.neucom.2017.11.022

Tzelepi, Maria; Tefas, Anastasios (2018), “Fully Unsupervised Convolutional Learning for Fast Image Retrieval”, [ACM Press the 10th Hellenic Conference - Patras, Greece (2018.07.09-2018.07.12)] Proceedings of the 10th Hellenic Conference on Artificial Intelligence - SETN '18, pp. 1–6. doi:10.1145/3200947.3201007

Chitrakar, Pradip; Zhang, Chengcui; Warner, Gary; Liao, Xinpeng (2016), “Social Media Image Retrieval Using Distilled Convolutional Neural Network for Suspicious e-Crime and Terrorist Account Detection”, [IEEE 2016 IEEE International Symposium on Multimedia (ISM) - San Jose, CA, USA (2016.12.11-2016.12.13)] 2016 IEEE International Symposium on Multimedia (ISM), pp.493–498. doi:10.1109/ISM.2016.0110

Zhuang, Fuzhen; Qi, Zhiyuan; Duan, Keyu; Xi, Dongbo; Zhu, Yongchun; Zhu, Hengshu; Xiong, Hui; He, Qing (2020), “ A Comprehensive Survey on Transfer Learning”, Proceedings of the IEEE, pp. 1–34. doi:10.1109/JPROC.2020.3004555

Day, O., & Khoshgoftaar, T. M. (2017), “A survey on heterogeneous transfer learning”, Journal of Big Data, 4(1). https://doi.org/10.1186/s40537-017-0089-0

Tan, C., Sun, F., Kong, T., Zhang, W., Yang, C., & Liu, C. (2018), “A Survey on Deep Transfer Learning”, Lecture Notes in Computer Science, 270–279. https://doi.org/10.1007/978-3-030-01424-7_27

Shorten, C., & Khoshgoftaar, T. M. (2019), “ A survey on Image Data Augmentation for Deep Learning”, Journal of Big Data, 6(1). 60-. https://doi.org/10.1186/s40537-019-0197-0

He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing; Sun, Jian (2016), “Deep Residual Learning for Image Recognition”, [IEEE 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) - Las Vegas, NV, USA (2016.6.27-2016.6.30)] 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.770–778. doi:10.1109/CVPR.2016.90

Szegedy, Christian; Wei Liu, ; Yangqing Jia, ; Sermanet, Pierre; Reed, Scott; Anguelov, Dragomir; Erhan, Dumitru; Vanhoucke, Vincent; Rabinovich, Andrew (2015), “Going deeper with convolutions”, [IEEE 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) - Boston, MA, USA (2015.6.7-2015.6.12)] 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). , pp. 1–9. doi:10.1109/CVPR.2015.7298594

Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017), “ImageNet classification with deep convolutional neural networks”, Communications of the ACM, 60(6), 84–90. https://doi.org/10.1145/3065386

Lu, Xin; Kang, Xin; Nishide, Shun; Ren, Fuji (2019), “Object detection based on SSD-ResNet”, [IEEE 2019 IEEE 6th International Conference on Cloud Computing and Intelligence Systems (CCIS) - Singapore (2019.12.19-2019.12.21)] 2019 IEEE 6th International Conference on Cloud Computing and Intelligence Systems (CCIS), 89–92. doi:10.1109/CCIS48116.2019.9073753

X. Liu, Y. Meng and M. Fu, "Classification Research Based on Residual Network for Hyperspectral Image," 2019 IEEE 4th International Conference on Signal and Image Processing (ICSIP), 2019, pp. 911-915, doi: 10.1109/SIPROCESS.2019.8868838.

Zhou, Yitao; Ren, Fuji; Nishide, Shun; Kang, Xin (2019), “Facial Sentiment Classification Based on Resnet-18 Model”, [IEEE 2019 International Conference on Electronic Engineering and Informatics (EEI) - Nanjing, China (2019.11.8-2019.11.10)] 2019 International Conference on Electronic Engineering and Informatics (EEI), 463–466. doi:10.1109/eei48997.2019.00106

Kumar, Vidit; Tripathi, Vikas; Pant, Bhaskar (2020), “Content based Fine-Grained Image Retrieval using Convolutional Neural Network”, [IEEE 2020 7th International Conference on Signal Processing and Integrated Networks (SPIN) - Noida, India (2020.2.27-2020.2.28)] 2020 7th International Conference on Signal Processing and Integrated Networks (SPIN), 1120–1125. doi:10.1109/SPIN48934.2020.9071334

Y. Tan, Y. Li, H. Liu, W. Lu and X. Xiao, (2020), “Performance Comparison of Data Classification based on Modern Convolutional Neural Network Architectures”, 2020 39th Chinese Control Conference (CCC), 2020, pp. 815-818, doi: 10.23919/CCC50068.2020.9189237.

Paidi, V., Fleyeh, H., & Nyberg, R. G. (2020), “Deep learning-based vehicle occupancy detection in an open parking lot using thermal camera”, IET Intelligent Transport Systems, 14(10), 1295–1302. https://doi.org/10.1049/iet-its.2019.0468

Malik, F., & Baharudin, B. (2013), “Analysis of distance metrics in content-based image retrieval using statistical quantized histogram texture features in the DCT domain”, Journal of King Saud University - Computer and Information Sciences, 25(2), 207–218. https://doi.org/10.1016/j.jksuci.2012.11.004

B. Patel, k. Yadav and D. Ghosh, (2020), “State-of-Art: Similarity Assessment for Content Based Image Retrieval System”, 2020 IEEE International Symposium on Sustainable Energy, Signal Processing and Cyber Security (iSSSC), 2020, pp. 1-6, doi: 10.1109/iSSSC50941.2020.9358899.

Ó Searcóid, Mícheál (2006), “2.7 Distances from Sets to Sets. Metric Spaces”, Springer Undergraduate Mathematics Series, Springer, pp. 29–30, ISBN 978-1-84628-627-8

Bellot, F., & Krause, E. E. (1988), “Taxicab Geometry: An Adventure in Non-Euclidean Geometry”, The Mathematical Gazette, 72(461), 255. https://doi.org/10.2307/3618288

Dengsheng Zhang, & Guojun Lu. (2003), “Evaluation of similarity measurement for image retrieval”, International Conference on Neural Networks and Signal Processing, 2003. Proceedings of the 2003, Vol.2, 928 - 931. https://doi.org/10.1109/icnnsp.2003.1280752

Wang, J. Z., Jia Li, & Wiederhold, G. (2001), “SIMPLIcity: semantics-sensitive integrated matching for picture libraries”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(9), 947–963. https://doi.org/10.1109/34.955109 https://sites.google.com/site/dctresearch/Home/content-based-image-retrieval.

Jia Li, & Wang, J. Z. (2003), “Automatic linguistic indexing of pictures by a statistical modeling approach”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(9), 1075–1088. https://doi.org/10.1109/tpami.2003.1227984

Kingma, D., and Jimmy, B. (2014), “ Adam: A Method for Stochastic Optimization”, arXiv preprint arXiv:1412.6980, 2014.

Robert, C. (2014), “Machine Learning, a Probabilistic Perspective”, CHANCE, 27(2), 62–63. https://doi.org/10.1080/09332480.2014.914768.

Li, H., Chaudhari, P., Yang, H., Lam, M., Ravichandran, A., Bhotika, R., Soatto, S., (2020), “Rethinking the Hyper-parameters for Fine-Tuning”, ICLR 2020, arXiv:2002.11770v1. https://doi.org/10.48550/arXiv.2002.11770

Zhou, S., & Song, W. (2020), “Deep learning-based roadway crack classification using laser-scanned range images: A comparative study on hyper-parameter selection”, Automation in Construction, 114, 103171. doi:10.1016/j.autcon.2020.103171

Adedigba, A. P., Adeshina, S. A., Aina, O. E., & Aibinu, A. M. (2021), “Optimal hyper-parameter selection of deep learning models for COVID-19 chest X-ray classification”, Intelligence-Based Medicine, 5, 100034. doi:10.1016/j.ibmed.2021.100034

Öztürk, Ş. (2021), “Convolutional neural network based dictionary learning to create hash codes for content-based image retrieval”, Procedia Computer Science, 183, 624–629. https://doi.org/10.1016/j.procs.2021.02.106

Trappey, A. J. C., Trappey, C. V., & Shih, S. (2021), “An intelligent content-based image retrieval methodology using transfer learning for digital IP protection”, Advanced Engineering Informatics, 48, 101291. https://doi.org/10.1016/j.aei.2021.101291

Pathak, D., & Raju, U. S. N. (2021), “Content-based image retrieval using feature-fusion of GroupNormalized-Inception-Darknet-53 features and handcraft features”, Optik, 246, 167754. https://doi.org/10.1016/j.ijleo.2021.167754

Zhang, N., Shamey, R., Xiang, J., Pan, R., Gao, W. (2022), “A novel image retrieval strategy based on transfer learning and hand-crafted features for wool fabric”, Expert Systems with Applications, 191, 116229. https://doi.org/10.1016/j.eswa.2021.116229

Framework of proposed CBIR system using classification scores




How to Cite

S. A. . Angadi and H. . C. Purad, “ Efficient Framework for Content-Based Image Retrieval using CNN Classification Scores”, Int J Intell Syst Appl Eng, vol. 11, no. 2, pp. 400–424, Feb. 2023.



Research Article