Multimodal Sentiment and Emotion Classification Using BiLSTM Model with Bird Intelligence and Bald Eagle Optimization
Keywords:
BiLSTM, Multimodal, White Headed Bird OptimizationAbstract
In the current digital era, when social media, customer reviews, and other online platforms generate massive amounts of text data every day, the ability to automatically identify and classify attitudes and emotions has become more and more important. Models of sentiment and emotion classification are vital tools for studying human emotional expression. Nevertheless, these models face many challenges, such as the need for effective multimodal data integration, handling of data imbalance, and the ability to capture nuanced emotional aspects. With the use of a white headed bird (WHB-) optimization and a bidirectional long short-term memory (BiLSTM) classifier, this work offers a strong approach for multimodal emotion classification. Furthermore, in order to improve overall efficiency, the paper presents a unique hybrid optimization technique that optimizes weights and biases in classifier parameters. The scientific novelty is in the combination of the cutting-edge WHB-optimization approach and the highly effective BiLSTM model, which together push the boundaries of multimodal emotional classification. By doing so, it makes a unique and beneficial addition to the field in the process. The created model achieves 94.90%, 95.27%, 94.79% and 95.78% accuracy performance for image, text, audio, and multimodal data, respectively.
Downloads
References
Huddar, Mahesh G., Sanjeev S. Sannakki, and Vijay S. Rajpurohit, “Attention-based multimodal contextual fusion for sentiment and emotion classification using bidirectional LSTM,” Multimedia Tools and Applications, vol. 80, no. 9, pp.13059-13076, 2021.
S. A. Abdu, A. H. Yousef, and A. Salem, “Multimodal video sentiment analysis using deep learning approaches, a survey,” Information Fusion (Print), vol. 76, pp. 204–226, Dec. 2021, doi: 10.1016/j.inffus.2021.06.003.
Maria Teresa Garcia-Ordas, Hector Alaiz-Moreton, Jose Alberto Benítez-Andrades, Isaias Garcia-Rodríguez, Oscar Garcia-Olalla, Carmen Benavides, “Sentiment analysis in non-fixed length audios using a Fully Convolutional Neural Network,” Biomedical Signal Processing and Control, Volume 69, p. 102946, Aug. 2021, doi: 10.1016/j.bspc.2021.102946.
Zhang, Dong, Shoushan Li,Qiaoming Zhu, and Guodong Zhou, “Multi-modal sentiment classification with independent and interactive knowledge via semi-supervised learning,” IEEE Access, vol. 8, pp. 22945-22954, 2020. doi: 10.1109/ACCESS.2020.2969205.
H. Kour and M. K. Gupta, “An hybrid deep learning approach for depression prediction from user tweets using feature-rich CNN and bi-directional LSTM,” Multimedia Tools and Applications, vol. 81, no. 17, pp. 23649–23685, Mar. 2022, doi: 10.1007/s11042-022-12648-y.
Hu Zhu, Ze Wang, Yu Shi, Yingying Hua, Guoxia Xu, and Lizhen Deng, “Multimodal fusion method based on self-attention mechanism,|” Wireless Communications and Mobile Computing 2020 (2020), 1–8. doi:10.1155/2020/8843186
Yang, Xiaocui, Shi Feng, Daling Wang, and Yifei Zhang, “Image-text multimodal emotion classification via multi-view attentional network,” IEEE Transactions on Multimedia, vol. 23 pp.4014-4026, 2020. doi:10.1109/TMM.2020.3035277
Huan Ruo-Hong, Jia Shu, Sheng-Lin Bao, Rong-Hua Liang, Peng Chen, and Kai-Kai Chi, “Video multimodal emotion recognition based on Bi-GRU and attention fusion,” Multimedia Tools and Applications, vol. 80, no. 6, pp. 8213-8240, 2021. doi: 10.1007/s11042-020-10030-4
Peng Wu, Xiaotong Li, Chen Ling, Shengchun Ding, Si Shen, “Sentiment classification using attention mechanism and bidirectional long short-term memory network,” Applied Soft Computing, Volume 112, pp. 107792, 2021. doi:10.1016/j.asoc.2021.107792.
Chen, Tian, Hongfang Yin, Xiaohui Yuan, Yu Gu, Fuji Ren, and Xiao Sun, “Emotion recognition based on fusion of long short-term memory networks and SVMs,” Digital Signal Processing, vol. 117, pp.103153, 2021.
Mai, Sijie, Haifeng Hu, Jia Xu, and Songlong Xing, “Multi-fusion residual memory network for multimodal human sentiment comprehension,” IEEE Transactions on Affective Computing, 2020.
Boateng, George, and Tobias Kowatsch, “Speech emotion recognition among elderly individuals using multimodal fusion and transfer learning,” In Companion Publication of the 2020 International Conference on Multimodal Interaction, pp. 12-16, 2020.
Lahat, Dana, TülayAdali, and Christian Jutten, “Multimodal data fusion: an overview of methods, challenges, and prospects,” Proceedings of the IEEE 103, no. 9, pp. 1449-1477, 2015.
Celli F, Lepri B, Biel J-I, Gatica-Perez D, Riccardi G, Pianesi F, The workshop on computational personality recognition, In: Proceedings of the 22nd ACM International Conference on Multimedia, Orlando, pp 1245–1246, 2014.
Poria S, Cambria E, Bajpai R, Hussain A, “A review of affective computing: from unimodal analysis to multimodal fusion,” Information Fusion, vol. 37, pp.:98–125, 2017.
[Morency LP, Mihalcea R, Doshi P, “Towards multimodal sentiment analysis: harvesting opinions from the web,” proceedings of the 13th international conference on multimodal interfaces, ICMI 2011, Alicante, Spain, pp. 14-18, 2011.
A. Zadeh, R. Zellers, E. Pincus, and L.-P. Morency, ‘‘Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages,’’ IEEE Intell. Syst., vol. 31, no. 6, pp. 82–88, Nov. 2016.
Zadeh, M. Chen, S. Poria, E. Cambria, and L. P. Morency, ‘‘Tensor fusion network for multimodal sentiment analysis,’’ in Proc. EMNLP, Copenhagen, Denmark, pp. 1103–1114, 2017.
S. Poria, E. Cambria, and A. Gelbukh, ‘‘Deep convolutional neural network textual features and multiple kernel learning for utterance-level multimodal sentiment analysis,’’ in Proc. Conf. Empirical Methods Natural Lang. Process., Lisbon, Portugal, pp. 2539–2544, 2015.
A. Zadeh, P. P. Liang, N. Mazumder, S. Poria, E. Cambria, and L. P. Morency, ‘‘Memory fusion network for multi-view sequential learning,’’ in Proc. AAAI, New Orleans, LA, USA, vol. 8, pp. 5634–5641,2020.
Leiva V, Freire A, “Towards suicide prevention: early detection of depression on social media,” International Conference on Internet Science. Springer, Cham, pp 428–436, 2017.
Stephen JJ, Prabu P, “Detecting the magnitude of depression in Twitter users using sentiment analysis,” International Journal of Electrical and Computer Engineering, vol. 9, no. 4, pp. 3247, 2019.
Bahdanau D, Cho K, Bengio Y, “Neural machine translation by jointly learning to align and translate”, pp. 1-15, 2014.
Busso C, Bulut M, Lee C, Kazemzadeh A, Mower E, Kim S, Chang J, Lee S, Narayanan S IEMOCAP: interactive emotional dyadic motion capture database. J Language ResourEvaluat, vol. 42, no. 4, pp. 335–359, 2008.
Cambria E, “Affective computing and sentiment analysis,” IEEE Intell Syst, vol. 31, no.2, pp.102–107, 2016.
Huddar, Mahesh G., Sanjeev S. Sannakki, and Vijay S. Rajpurohit, “Attention-based word-level contextual feature extraction and cross-modality fusion for sentiment analysis and emotion classification,” International Journal of Intelligent Engineering Informatics, vol. 8, no. 1, pp.1-18, 2020.
Zhang, Yazhou, Prayag Tiwari, Lu Rong, Rui Chen, Nojoom A.AlNajem, and M. Shamim Hossain, “Affective Interaction: Attentive Representation Learning for Multi-Modal Sentiment Classification,” ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 2021.
Huddar, Mahesh G., Sanjeev S. Sannakki, and Vijay S. Rajpurohit, “Multi-level context extraction and attention-based contextual inter-modal fusion for multimodal sentiment analysis and emotion classification,” International Journal of Multimedia Information Retrieval, vol. 9, no. 2, pp. 103-112, 2020.
Multimodal emotion recognition dataset, MVSA – Single from https://www.kaggle.com/datasets/vincemarcs/mvsasingle
[30] Multimodal sentiment classification dataset, MVSA – Multiple from https://www.kaggle.com/datasets/vincemarcs/mvsamultiple
Mishra, Kaushik, and Santosh Kumar Majhi, “A binary bird swarm optimization based load balancing algorithm for cloud computing environment,” Open Computer Science, vol. 11, no. 1 pp. 146-160, 2021.
Sayed, Gehad Ismail, Mona M. Soliman, and Aboul Ella Hassanien, “A novel melanoma prediction model for imbalanced data using optimized SqueezeNet by bald eagle search optimization,” Computers in Biology and Medicine, vol. 136, pp. 104712, 2021.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.
IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
