Multimodal Sentiment Analysis using Multiple Neural Networks and Natural Language Processing Models

Abstract

Multimodal Sentiment Analysis (MSA) identifies emotional expressions over time using visual and audio information. MSA, an emerging field, detects and analyzes emotions across text, speech, and images, with practical applications in social media analysis, customer feedback analysis, healthcare monitoring, and investigations. In the realm of e-commerce, product reviews significantly influence consumer purchasing behaviours, with user-generated content offering diverse perspectives on products and services. Sentiment analysis systematically deciphers the emotional undertones within text, providing businesses with actionable insights. Videos and audio recordings introduce richer dimensions to these reviews, capturing nuanced expressions and vocal cues. Integrating both modalities into sentiment analysis offers a deeper comprehension of reviewers' emotional states, enhancing the accuracy of product perception and satisfaction assessments.

Our innovative approach merges the Haar cascade algorithm for facial detection and sentiment analysis in videos with the BERT model for audio sentiment analysis, synergizing visual and auditory cues. This integration improves accuracy and robustness, facilitating precise assessments of product sentiment. By fusing outputs from these modalities, we can identify the most salient emotional cues expressed by reviewers, providing businesses with a comprehensive understanding of consumer sentiment and aiding informed decision-making processes.

This paper presents a novel approach to sentiment analysis using facial expression and audio data from the CMU-MOSI dataset. The process involves extracting a focused video of the subject's face, converting the corresponding audio into text, and analyzing sentiment. The proposed methods achieved promising accuracy and hold immense potential when combined with other modalities.