An Efficient Model for Visual Sentiment Analysis using Hybrid Feature Extraction and Fusion-Based Classification
Keywords:
Visual Sentiment Analysis, Visual Features, Convolutional Neural NetworksAbstract
The exponential growth of the internet technology industry has led users to share their opinions/sentiments through an online platform not only in the form of text but also in the form of images, speech, and videos in variety of applications. Researchers are focusing on building a sentiment analysis model based on image data, as it offers a more effective method for analyzing sentiments. The need is addressed with a model for visual sentiment analysis which uses hybrid feature extraction and fusion-based classification method The proposed approach strategically combines the strengths of multiple visual features, selected through the effective Dual Moth Flame Optimization (DMFO) model. To effectively leverage the selected features, a customized Fusion-based Convolutional Neural Network (CNN) architecture is specifically designed for visual analysis for multiclass sentiment classification, categorizing visual sentiments into positive, negative, and neutral. Our proposed model is superior to existing approaches, as shown by empirical evaluations on multiple datasets and achieves outperforming efficiency as compared to existing methods. In addition, the model's applicability to real-time scenarios is promising. The approach ensures robust performance and holds promise for applications in social media analysis, marketing, user experience assessment which increases the model's adaptability levels.
Downloads
References
Xue, L. Y., Mao, Q. R., Huang, X. H., & Chen, J. (2020). NLWSNet: a weakly supervised network for visual sentiment analysis in mislabeled web images. Frontiers of Information Technology & Electronic Engineering, 21(9), 1321-1333.
Mittal, N., Sharma, D., & Joshi, M. L. (2018, December). Image sentiment analysis using deep learning. In 2018 IEEE/WIC/ACM international conference on web intelligence (WI) (pp. 684-687). IEEE.
Ortis, A., Farinella, G. M., & Battiato, S. (2020). Survey on visual sentiment analysis. IET Image Processing, 14(8), 1440-1456.
She, D., Yang, J., Cheng, M. M., Lai, Y. K., Rosin, P. L., & Wang, L. (2019). Wscnet: Weakly supervised coupled networks for visual sentiment classification and detection. IEEE Transactions on Multimedia, 22(5), 1358-1371.
J. Xu, Z. Li, F. Huang, C. Li and P. S. Yu, "Visual Sentiment Analysis With Social Relations-Guided Multiattention Networks," in IEEE Transactions on Cybernetics, vol. 52, no. 6, pp. 4472-4484, June 2022, doi: 10.1109/TCYB.2020.3027766.
Chen, J., Mao, Q., & Xue, L. (2020). Visual sentiment analysis with active learning. IEEE Access, 8, 185899-185908.
Mamatha, M., Shivakumar, S., Thriveni, J., & Venugopal, K. R. (2022, July). Visual Sentiment Classification of Restaurant Review Images using Deep Convolutional Neural Networks. In 2022 IEEE International Conference on Electronics, Computing and Communication Technologies (CONECCT) (pp. 1-6). IEEE.
S. Urolagin, J. Nayak and U. R. Acharya, "Gabor CNN Based Intelligent System for Visual Sentiment Analysis of Social Media Data on Cloud Environment," in IEEE Access, vol. 10, pp. 132455-132471, 2022, doi: 10.1109/ACCESS.2022.3228263.
Wu, Z., Meng, M., & Wu, J. (2020). Visual sentiment prediction with attribute augmentation and multi-attention mechanism. Neural Processing Letters, 51, 2403-2416.
Song, K., Yao, T., Ling, Q., & Mei, T. (2018). Boosting image sentiment analysis with visual attention. Neurocomputing, 312, 218-228.
T. Zhou, J. Cao, X. Zhu, B. Liu and S. Li, "Visual-Textual Sentiment Analysis Enhanced by Hierarchical Cross-Modality Interaction," in IEEE Systems Journal, vol. 15, no. 3, pp. 4303-4314, Sept. 2021, doi: 10.1109/JSYST.2020.3026879.
S. Lee, D. K. Han and H. Ko, "Multimodal Emotion Recognition Fusion Analysis Adapting BERT With Heterogeneous Feature Unification," in IEEE Access, vol. 9, pp. 94557-94572, 2021, doi: 10.1109/ACCESS.2021.3092735.
Truong, Q. T., & Lauw, H. W. (2017, October). Visual sentiment analysis for review images with item-oriented and user-oriented CNN. In Proceedings of the 25th ACM international conference on Multimedia (pp. 1274-1282).
Meena, G., Mohbey, K. K., & Kumar, S. (2023). Sentiment analysis on images using convolutional neural networks based Inception-V3 transfer learning approach. International Journal of Information Management Data Insights, 3(1), 100174.
J. Zhang, X. Liu, Z. Wang and H. Yang, "Graph-Based Object Semantic Refinement for Visual Emotion Recognition," in IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 5, pp. 3036-3049, May 2022, doi: 10.1109/TCSVT.2021.3098712
Boughida, A., Kouahla, M. N., & Lafifi, Y. (2022). A novel approach for facial expression recognition based on Gabor filters and genetic algorithm. Evolving Systems, 13(2), 331-345.
Fallahzadeh, M. R., Farokhi, F., Harimi, A., & Sabbaghi-Nadooshan, R. (2021). Facial expression recognition based on image gradient and deep convolutional neural network. Journal of AI and Data Mining, 9(2), 259-268.
Hua, W., Dai, F., Huang, L., Xiong, J., & Gui, G. (2019). HERO: Human emotions recognition for realizing intelligent Internet of Things. IEEE Access, 7, 24321-24332.
Agrawal, A., & Mittal, N. (2020). Using CNN for facial expression recognition: a study of the effects of kernel size and number of filters on accuracy. The Visual Computer, 36(2), 405-412
F. Alzamzami and A. E. Saddik, "Transformer-Based Feature Fusion Approach for Multimodal Visual Sentiment Recognition Using Tweets in the Wild," in IEEE Access, vol. 11, pp. 47070-47079, 2023, doi: 10.1109/ACCESS.2023.3274744.
S. Urolagin, J. Nayak and U. R. Acharya, "Gabor CNN Based Intelligent System for Visual Sentiment Analysis of Social Media Data on Cloud Environment," in IEEE Access, vol. 10, pp. 132455-132471, 2022, doi: 10.1109/ACCESS.2022.3228263.
Y. Su, W. Zhao, P. Jing and L. Nie, "Exploiting Low-Rank Latent Gaussian Graphical Model Estimation for Visual Sentiment Distributions," in IEEE Transactions on Multimedia, vol. 25, pp. 1243-1255, 2023, doi: 10.1109/TMM.2022.3140892.
J. Kossaifi et al., "SEWA DB: A Rich Database for Audio-Visual Emotion and Sentiment Research in the Wild," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 3, pp. 1022-1040, 1 March 2021, doi: 10.1109/TPAMI.2019.2944808.
L. Stappen, A. Baird, L. Schumann and B. Schuller, "The Multimodal Sentiment Analysis in Car Reviews (MuSe-CaR) Dataset: Collection, Insights and Improvements," in IEEE Transactions on Affective Computing, vol. 14, no. 2, pp. 1334-1350, 1 April-June 2023, doi: 10.1109/TAFFC.2021.3097002.
J. He, S. Mai and H. Hu, "A Unimodal Reinforced Transformer With Time Squeeze Fusion for Multimodal Sentiment Analysis," in IEEE Signal Processing Letters, vol. 28, pp. 992-996, 2021, doi: 10.1109/LSP.2021.3078074.
Qayyum, H., Majid, M., Anwar, S. M., & Khan, B. (2017). Facial expression recognition using stationary wavelet transform features. Mathematical Problems in Engineering, 2017.
Avcı, D., Sert, E., Özyurt, F., & Avcı, E. (2024). MFIF-DWT-CNN: Multi-focus ımage fusion based on discrete wavelet transform with deep convolutional neural network. Multimedia Tools and Applications, 83(4), 10951-10968..
You, N., Han, L., Zhu, D., & Song, W. (2023). Research on image denoising in edge detection based on wavelet transform. Applied Sciences, 13(3), 1837.
J. Zeng, J. Zhou and C. Huang, "Exploring Semantic Relations for Social Media Sentiment Analysis," in IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 2382-2394, 2023, doi: 10.1109/TASLP.2023.3285238.
K. Ye, N. H. Nazari, J. Hahn, Z. Hussain, M. Zhang and A. Kovashka, "Interpreting the Rhetoric of Visual Advertisements," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 4, pp. 1308-1323, 1 April 2021, doi: 10.1109/TPAMI.2019.2947440.
Truong, Q. T., & Lauw, H. W. (2023, February). Concept-oriented transformers for visual sentiment analysis. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining (pp. 1111-1119).
Chatterjee, S., Saha, D., Sen, S., Oliva, D., & Sarkar, R. (2023). Moth-flame optimization based deep feature selection for facial expression recognition using thermal images. Multimedia Tools and Applications, 1-24.
Mirjalili, S. (2015). Moth-flame optimization algorithm: A novel nature-inspired heuristic paradigm. Knowledge-based systems, 89, 228-249.
Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010, June). The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In 2010 ieee computer society conference on computer vision and pattern recognition-workshops (pp. 94-101). IEEE.
Akhand, M. A. H., Roy, S., Siddique, N., Kamal, M. A. S., & Shimamura, T. (2021). Facial emotion recognition using transfer learning in the deep CNN. Electronics, 10(9), 1036.
Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., ... & Bengio, Y. (2013). Challenges in representation learning: A report on three machine learning contests. In Neural Information Processing: 20th International Conference, ICONIP 2013, Daegu, Korea, November 3-7, 2013. Proceedings, Part III 20 (pp. 117-124). Springer berlin heidelberg.
Deshmukh, P. S., Gaikwad, A. K.., & Agrawal P. K. (2024). Deep Learning Based Dual-level Bioinspired Model for Parkinson’s Disease Detection. International Journal of Intelligent Systems and Applications in Engineering, 12(13s), 179–187.
Agrawal, P.K. A Novel Mapper Machine Learning Algorithm for Semantic Domain Mapping for Domain Database Updation. SN COMPUT. SCI. 4, 536 (2023).
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.
IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
