Improving the personalization of the EMASPEL learning experience through deep learning-based facial expression recognition
Keywords:
Facial Expression Recognition, ITS, FERAbstract
Conventional Intelligent Tutoring Systems (ITS) rely solely on cognitive performance metrics to personalize learning journeys, overlooking the crucial role of emotions in facilitating effective learning. This disconnects often results in disengaged students and suboptimal learning outcomes. Our research presents a novel approach to bridge this gap by integrating Deep Learning-based Facial Expression Recognition (FER) into the EMASPEL ITS. We propose utilizing FER to equip EMASPEL with the ability to provide immediate feedback on students' engagement and emotional states. Our focus lies in identifying key emotions such as annoyance, confusion, and excitement, using the power of Deep Learning algorithms to accurately interpret facial expressions. This real-time emotional understanding empowers EMASPEL to dynamically adjust educational content and pace in sync with students' emotional responses. We acknowledge the challenges and ethical considerations surrounding the implementation of FER in educational settings. Transparency and robust privacy measures are paramount in ensuring this technology is utilized responsibly and ethically. We envision EMASPEL as a pioneer in fostering emotionally aware ITS, not just catering to cognitive needs but catering to the entire spectrum of human emotions. Our contribution lies in establishing a framework for developing advanced ITS that leverage Deep Learning-powered FER to recognize and respond to a wide range of human emotions. This paves the way for truly personalized learning experiences that prioritize engagement, motivation, and ultimately, maximize educational achievement.
Downloads
References
Kasthurirangan Gopalakrishnan, Siddhartha K Khaitan, Alok Choudhary, and Ankit Agrawal. (2017). Deep convolutional neural networks with transfer learning for computer vision-based data-driven pavement distress detection. Construction and Building Materials, 157, 322–330.
Mahmoud Neji, Mohamed Ben Ammar, and Guy Gouardères. (2007). Affective Communication for Peer-to-Peer e-Learning. In The International Conference on Computing and e-Systems, Hammamet, Tunisia, March 12-14 (pp. 252–268).
Nouha Khediri, Mohamed Ben Ammar, and Monji Kherallah. (2021). Comparison of image segmentation using different color spaces. In 2021 IEEE 21st International Conference on Communication Technology (ICCT) (pp. 1188–1192).
Aya Hassouneh, A.M. Mutawa, and M. Murugappan. (2020). Development of a real-time emotion recognition system using facial expressions and EEG based on machine learning and deep neural network methods. Informatics in Medicine Unlocked, 20, 100372.
Tong Yu and Hong Zhu. (2020). Hyper-parameter optimization: A review of algorithms and applications. arXiv preprint arXiv:2003.05689.
Manoj Moolchandani, Shivangi Dwivedi, Samarth Nigam, and Kapil Gupta. (2021). A survey on: Facial emotion recognition and classification. In 2021 5th International Conference on Computing Methodologies and Communication (ICCMC) (pp. 1677–1686).
Felipe Zago Canal, Tobias Rossi Müller, Jhennifer Cristine Matias, Gustavo Gino Scotton, Antonio Reis de Sa Junior, Eliane Pozzebon, and Antonio Carlos Sobieranski. (2022). A survey on facial emotion recognition techniques: A state-of-the-art literature review. Information Sciences, 582, 593–617.
Megha Singh, Shiva Kant Sharma, Shouhaddo Paul, Jithin P Sajeevan, and Sandarya Paul. (2020). Facial emotion recognition system. Journal of Scientific Research and Advances, 6(6).
Aakash Saroop, Pathik Ghugare, Sashank Mathamsetty, and Vaibhav Vasani. (2021). Facial emotion recognition: A multi-task approach using deep learning. CoRR, abs/2110.15028.
Shervin Minaee and Amirali Abdolrashidi. (2019). Deep-emotion: Facial expression recognition using attentional convolutional network.
Matthew D. Zeiler and Rob Fergus. (2013). Visualizing and understanding convolutional networks. CoRR, abs/1311.2901.
Lining Wang, Zheng He, Bin Meng, Kai Liu, Qingyu Dou, and Xiaomin Yang. (2021). Two-pathway attention network for real-time facial expression recognition. Journal of Real-Time Image Processing, 18(4), 1173–1182.
Koray U. Erbas. (2021). Facial emotion recognition with convolutional neural network based architecture. International Journal of Computer and Information Engineering, 15(1), 67–74.
M. Amine Mahmoudi, Aladine Chetouani, Fatma Boufera, and Hedi Tabia. (2020). Improved Bilinear Model for Facial Expression Recognition. In 4th Mediterranean Conference on Pattern Recognition and Artificial Intelligence (MedPRAI 2020) (Vol. 1322, pp. 47–59).
Zhenxing Gao, Ben Niu, and Bingbing Guo. (2021). Facial expression recognition with LBP and ORB features. Computational Intelligence and Neuroscience.
Yue Wu, Hao Meng, Fei Yuan, and Tianhao Yan. (2021). Facial expression recognition algorithm based on fusion of transformed multilevel features and improved weighted voting SVM. Mathematical Problems in Engineering, 1–117.
Simonyan and Andrew Zisserman. (2015). Very deep convolutional networks for large-scale image recognition. In 3rd International Conference on Learning Representations, ICLR 2015.
Imane Lasri, Anouar Riad Solh, and Mourad El Belkacemi. (2019). Facial emotion recognition of students using convolutional neural network. In 2019 Third International Conference on Intelligent Computing in Data Sciences (ICDS) (pp. 1–6).
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. (2009). ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition (pp. 248–255).
Ahmad Waleed Salehi, Shakir Khan, Gaurav Gupta, Bayan Ibrahimm Alabduallah, Abrar Almjally, Hadeel Alsolai, Tamanna Siddiqui, and Adel Mellit. (2023). A study of CNN and transfer learning in medical imaging: Advantages, challenges, future scope. Sustainability, 15(7), 5930.
Vinod Nair and Geoffrey E Hinton. (2010). Rectified linear units improve restricted Boltzmann machines. In ICML 2010 (pp. 807–814).
Diederik P Kingma and Jimmy Ba. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Patrick Lucey, Jeffrey F. Cohn, Takeo Kanade, Jason Saragih, Zara Ambadar, and Iain Matthews. (2010). The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops (pp. 94–101).
MAH Akhand, Shuvendu Roy, Nazmul Siddique, Md Abdus Samad Kamal, and Tetsuya Shimamura. (2021). Facial emotion recognition using transfer learning in the deep CNN. Electronics, 10(9), 1036.
Veena Mayya, Radhika M Pai, and MM Manohara Pai. (2016). Automatic facial expression recognition using DCNN. Procedia Computer Science, 93, 453–461.
Rio Febrian, Benedic Matthew Halim, Maria Christina, Dimas Ramdhan, and Andry Chowanda. (2023). Facial expression recognition using bidirectional LSTM-CNN. Procedia Computer Science, 216, 39–47.
Dandan Liang, Huagang Liang, Zhenbo Yu, and Yipu Zhang. (2020). Deep convolutional BiLSTM fusion network for facial expression recognition. The Visual Computer, 36, 499–508.
G. Caridakis, J. Wagner, A. Raouzaiou, Z. Curto, E. Andre, K. Karpouzis. (2010). A multimodal corpus for gesture expressivity analysis. Image, Video and Multimedia Systems, Laboratory National Technical University of Athens Iroon Polytexneiou 9, 15780 Zografou, Greece.
Celso M. de Melo, Jonathan Gratch, Stacy Marsella, Catherine Pelachaud. (2023). Social Functions of Machine Emotional Expressions. Proc. IEEE, 111(10), 1382–1397.
Vladislav Maraev, Chiara Mazzocconi, Christine Howes, Catherine Pelachaud. (2023). Towards investigating gaze and laughter coordination in socially interactive agents. HAI 2023, 473–475.
Silèye O. Ba and Jean-Marc Odobez. (2010). Multi-Person Visual Focus of Attention from Head Pose and Meeting Contextual Cues. IEEE Trans. on Pattern Analysis and Machine Intelligence.
Beyan, C., Vinciarelli, A., & Bue, A. D. (2023). Co-Located Human–Human Interaction Analysis Using Nonverbal Cues: A Survey. ACM Computing Surveys, 56(5), 1–41.
Ahmed, N., Al Aghbari, Z., & Girija, S. (2023). A systematic survey on multimodal emotion recognition using learning algorithms. Intelligent Systems with Applications, 17, 200171.
Li, Y. K., Meng, Q. H., Wang, Y. X., & Hou, H. R. (2023). MMFN: Emotion recognition by fusing touch gesture and facial expression information. Expert Systems with Applications, 228, 120469.
Bhaumik, G., Verma, M., Govil, M. C., & Vipparthi, S. K. (2023). Hyfinet: Hybrid feature attention network for hand gesture recognition. Multimedia Tools and Applications, 82(4), 4863–4882.
Boyali, A., & Hashimoto, N. (2023). Hand Posture Control of a Robotic Wheelchair Using a Leap Motion Sensor and Block Sparse Representative Classification Method.
Zhang, X., Fan, J., Peng, T., Zheng, P., Zhang, X., & Tang, R. (2023). Multimodal data-based deep learning model for sitting posture recognition toward office workers’ health promotion. Sensors and Actuators A: Physical, 350, 114150.
S. Gutta, J. Huang, I.F. Imam, and H. Wechsler. (1996). Face and hand gesture recognition using hybrid classifiers. Technical report.
J.L Crowley and J. Martin. (1997). Visual processes for tracking and recognition of hand gestures. In Workshop on Perceptual User Interfaces (PUI’97).
Miah, A. S. M., Hasan, M. A. M., & Shin, J. (2023). Dynamic Hand Gesture Recognition using Multi-Branch Attention Based Graph and General Deep Learning Model. IEEE Access, 11, 4703–4716.
David B. Givens. (2010). The nonverbal dictionary of Gestures, Signs & Body Language Cues. Spokane, Washington: Center for Nonverbal Studies Press.
Pan, B., Hirota, K., Jia, Z., Zhao, L., Jin, X., & Dai, Y. (2023). Multimodal emotion recognition based on feature selection and extreme learning machine in video clips. Journal of Ambient Intelligence and Humanized Computing, 14(3), 1903–1917.
Shi, H. (2023). Learning-based human action and affective gesture analysis.
Deng, H., Yang, Z., Hao, T., Li, Q., & Liu, W. (2022). Multimodal Affective Computing with Dense Fusion Transformer for Inter-and Intra-modality Interactions. IEEE Transactions on Multimedia.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.
IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
