Facial Expression Recognition Using Expression Generative Adversarial Network and Attention CNN
Keywords:
Facial Emotion Recognition, Generative Adversarial Network, Convolutional Neural NetworkAbstract
Facial Expressions are quite personalized and may look different for different individuals. Whereas there are certain facial muscles which shows some common features for certain human expressions across the cultures and facial shapes. Convolution Neural Network have shown tremendous success in Facial Expression Recognition task. In recent past many researchers have proposed multiple models with manageable size solution to Facial Expression Recognition task. In the current work, we have considered shape, complexion and other identity related information separate from certain specified muscle movements which are specific for emotion recognition. This is done by a novel Emotion-Generative Adversarial Network. This saves a lot of effort and simplifies the Facial Expression Recognition process. We then apply Scale Invariant Feature Transformation and vola john’s face extraction method for pre-processing and face image extraction from background. This enables us to train our model accurately irrespective of scale, orientation, illumination etc and with very less training samples accurately. We feed the feature extracted facial image to an attention-based Convolutional Neural Network. This will ensure more emphasis on critical areas for expression recognition of facial image. Finally, we have used Local Binary Pattern for classification of the input image to a particular emotion class. We have tested our model on CK+, OULU- Casia and FER-2013 datasets and it is at par with performance of all major state-of-art models. Proposed model may be utilized by various automated interactive systems, such as robot to human communication, automated customer care systems etc. The proposed work may also be quite useful for observing reaction of viewers to a particular advertisement or article automatically and use this information for various purposes like user’s interest, product feedback etc.
Downloads
References
R. Kosti, J. M. Alvarez, A. Recasens, and A. Lapedriza, “Emotion Recognition in Context.”
J. Li, K. Jin, D. Zhou, N. Kubota, and Z. Ju, “Attention mechanism-based CNN for facial expression recognition,” Neurocomputing, vol. 411, pp. 340–350, Oct. 2020, doi: 10.1016/j.neucom.2020.06.014.
P. P. Angelov, IEEE Computational Intelligence Society, International Neural Network Society. Morocco Regional Chapter, and Institute of Electrical and Electronics Engineers, ICDS2019 : the Third International Conference on Intelligent Computing in Data Sciences : October 28-29-30, 2019, Marrakech, Morocco.
F. Wang, J. Lv, G. Ying, S. Chen, and C. Zhang, “Facial expression recognition from image based on hybrid features understanding,” J Vis Commun Image Represent, vol. 59, pp. 84–88, Feb. 2019, doi: 10.1016/j.jvcir.2018.11.010.
H. Li, J. Sun, Z. Xu, and L. Chen, “Multimodal 2D+3D Facial Expression Recognition with Deep Fusion Convolutional Neural Network,” IEEE Trans Multimedia, vol. 19, no. 12, pp. 2816–2831, Dec. 2017, doi: 10.1109/TMM.2017.2713408.
P. Isola, J. Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Institute of Electrical and Electronics Engineers Inc., Nov. 2017, pp. 5967–5976. doi: 10.1109/CVPR.2017.632.
B. Yang, J. Cao, R. Ni, and Y. Zhang, “Facial Expression Recognition Using Weighted Mixture Deep Neural Network Based on Double-Channel Facial Images,” IEEE Access, vol. 6, pp. 4630–4640, Dec. 2017, doi: 10.1109/ACCESS.2017.2784096.
W. Mellouk and W. Handouzi, “Facial emotion recognition using deep learning: Review and insights,” in Procedia Computer Science, Elsevier B.V., 2020, pp. 689–694. doi: 10.1016/j.procs.2020.07.101.
T. Connie, M. Al-Shabi, W. P. Cheah, and M. Goh, “Facial expression recognition using a hybrid CNN-SIFT aggregator,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Springer Verlag, 2017, pp. 139–149. doi: 10.1007/978-3-319-69456-6_12.
Y. Xia, W. Zheng, Y. Wang, H. Yu, J. Dong, and F. Y. Wang, “Local and Global Perception Generative Adversarial Network for Facial Expression Synthesis,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 3, pp. 1443–1452, Mar. 2022, doi: 10.1109/TCSVT.2021.3074032.
I. Lasri, A. R. Solh, and M. El Belkacemi, “Facial Emotion Recognition of Students using Convolutional Neural Network,” in 2019 Third International Conference on Intelligent Computing in Data Sciences (ICDS), IEEE, Oct. 2019, pp. 1–6. doi: 10.1109/ICDS47004.2019.8942386.
Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo, “StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation,” Nov. 2017, [Online]. Available: http://arxiv.org/abs/1711.09020
V. Sreenivas, V. Namdeo, and E. Vijay Kumar, “Modified deep belief network based human emotion recognition with multiscale features from video sequences,” Softw Pract Exp, vol. 51, no. 6, pp. 1259–1279, Jun. 2021, doi: 10.1002/spe.2955.
S. E. Kahou et al., “EmoNets: Multimodal deep learning approaches for emotion recognition in video,” Journal on Multimodal User Interfaces, vol. 10, no. 2, pp. 99–111, Jun. 2016, doi: 10.1007/s12193-015-0195-2.
S. Li and W. Deng, “Blended Emotion in-the-Wild: Multi-label Facial Expression Recognition Using Crowdsourced Annotations and Deep Locality Feature Learning,” Int J Comput Vis, vol. 127, no. 6–7, pp. 884–906, Jun. 2019, doi: 10.1007/s11263-018-1131-1.
B. C. Ko, “A brief review of facial emotion recognition based on visual information,” Sensors (Switzerland), vol. 18, no. 2, Feb. 2018, doi: 10.3390/s18020401.
D. Y. Liliana, “Emotion recognition from facial expression using deep convolutional neural network,” in Journal of Physics: Conference Series, Institute of Physics Publishing, Apr. 2019. doi: 10.1088/1742-6596/1193/1/012004.
C. Marechal et al., “Survey on AI-based multimodal methods for emotion detection,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Springer Verlag, 2019, pp. 307–324. doi: 10.1007/978-3-030-16272-6_11.
W. Huang, S. Zhang, P. Zhang, Y. Zha, Y. Fang, and Y. Zhang, “Identity-Aware Facial Expression Recognition Via Deep Metric Learning Based on Synthesized Images,” IEEE Trans Multimedia, vol. 24, pp. 3327–3339, 2022, doi: 10.1109/TMM.2021.3096068.
S. Li and W. Deng, “Deep Facial Expression Recognition: A Survey,” Apr. 2018, doi: 10.1109/TAFFC.2020.2981446.
B. Martinez, M. F. Valstar, B. Jiang, and M. Pantic, “Automatic analysis of facial actions: A survey,” IEEE Transactions on Affective Computing, vol. 10, no. 3. Institute of Electrical and Electronics Engineers Inc., pp. 325–347, Jul. 25, 2019. doi: 10.1109/TAFFC.2017.2731763.
Davis, W., Wilson, D., López, A., Gonzalez, L., & González, F. Automated Assessment and Feedback Systems in Engineering Education: A Machine Learning Approach. Kuwait Journal of Machine Learning, 1(1). Retrieved from http://kuwaitjournals.com/index.php/kjml/article/view/102
S. Xie, H. Hu, and Y. Chen, “Facial Expression Recognition with Two-Branch Disentangled Generative Adversarial Network,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, no. 6, pp. 2359–2371, Jun. 2021, doi: 10.1109/TCSVT.2020.3024201.
I. Yildirim, M. Belledonne, W. Freiwald, and J. Tenenbaum, “Efficient inverse graphics in biological face processing,” 2020.
G. Ali et al., “Artificial Neural Network Based Ensemble Approach for Multicultural Facial Expressions Analysis,” IEEE Access, vol. 8, pp. 134950–134963, 2020, doi: 10.1109/ACCESS.2020.3009908.
A. Gupta, S. Arunachalam, and R. Balakrishnan, “Deep self-attention network for facial emotion recognition,” in Procedia Computer Science, Elsevier B.V., 2020, pp. 1527–1534. doi: 10.1016/j.procs.2020.04.163.
D. Liu, L. Wang, Z. Wang, and L. Chen, “Novel multi‐scale deep residual attention network for facial expression recognition,” The Journal of Engineering, vol. 2020, no. 12, pp. 1220–1226, Dec. 2020, doi: 10.1049/joe.2020.0183.
S. Rajan, P. Chenniappan, S. Devaraj, and N. Madian, “Novel deep learning model for facial expression recognition based on maximum boosted CNN and LSTM,” IET Image Process, vol. 14, no. 7, pp. 1227–1232, May 2020, doi: 10.1049/iet-ipr.2019.1188.
T. H. Vo, G. S. Lee, H. J. Yang, and S. H. Kim, “Pyramid with Super Resolution for In-the-Wild Facial Expression Recognition,” IEEE Access, vol. 8, pp. 131988–132001, 2020, doi: 10.1109/ACCESS.2020.3010018.
H. Li and H. Xu, “Deep reinforcement learning for robust emotional classification in facial expression recognition,” Knowl Based Syst, vol. 204, Sep. 2020, doi: 10.1016/j.knosys.2020.106172.
A. John, M. C. Abhishek, A. S. Ajayan, S. Sanoop, and V. R. Kumar, “Real-time facial emotion recognition system with improved preprocessing and feature extraction,” in Proceedings of the 3rd International Conference on Smart Systems and Inventive Technology, ICSSIT 2020, Institute of Electrical and Electronics Engineers Inc., Aug. 2020, pp. 1328–1333. doi: 10.1109/ICSSIT48917.2020.9214207.
N. Mehendale, “Facial emotion recognition using convolutional neural networks (FERC),” SN Appl Sci, vol. 2, no. 3, Mar. 2020, doi: 10.1007/s42452-020-2234-1.
G. P. R, E. Granger, and P. Cardinal, “Deep Domain Adaptation for Ordinal Regression of Pain Intensity Estimation Using Weakly-Labelled Videos,” Aug. 2020, [Online]. Available: http://arxiv.org/abs/2008.06392
J. Lee, S. Kim, S. Kim, and K. Sohn, “Multi-Modal Recurrent Attention Networks for Facial Expression Recognition,” IEEE Transactions on Image Processing, vol. 29, pp. 6977–6991, 2020, doi: 10.1109/TIP.2020.2996086.
S. M. S. A. Abdullah, S. Y. A. Ameen, M. A. M. Sadeeq, and S. Zeebaree, “Multimodal Emotion Recognition using Deep Learning,” Journal of Applied Science and Technology Trends, vol. 2, no. 02, pp. 52–58, Apr. 2021, doi: 10.38094/jastt20291.
A. Barman and P. Dutta, “Facial expression recognition using distance and shape signature features,” Pattern Recognit Lett, vol. 145, pp. 254–261, May 2021, doi: 10.1016/j.patrec.2017.06.018.
D. G. R. Kola and S. K. Samayamantula, “Facial expression recognition using singular values and wavelet-based LGC-HD operator,” IET Biom, vol. 10, no. 2, pp. 207–218, Mar. 2021, doi: 10.1049/bme2.12012.
S. Saurav, R. Saini, and S. Singh, “EmNet: a deep integrated convolutional neural network for facial emotion recognition in the wild,” Applied Intelligence, vol. 51, no. 8, pp. 5543–5570, Aug. 2021, doi: 10.1007/s10489-020-02125-0.
Kumar, C. ., & Muthumanickam, T. . (2023). Analysis of Unmanned Four-Wheeled Bot with AI Evaluation Feedback Linearization Method. International Journal on Recent and Innovation Trends in Computing and Communication, 11(2), 138–142. https://doi.org/10.17762/ijritcc.v11i2.6138
F. Zhang, T. Zhang, Q. Mao, and C. Xu, “A Unified Deep Model for Joint Facial Expression Recognition, Face Synthesis, and Face Alignment,” IEEE Transactions on Image Processing, vol. 29, pp. 6574–6589, 2020, doi: 10.1109/TIP.2020.2991549.
J. Cai et al., “IDENTITY-FREE FACIAL EXPRESSION RECOGNITION USING CONDITIONAL GENERATIVE ADVERSARIAL NETWORK,” in Proceedings - International Conference on Image Processing, ICIP, IEEE Computer Society, 2021, pp. 1344–1348. doi: 10.1109/ICIP42928.2021.9506593.
Y. Yan, Y. Huang, S. Chen, C. Shen, and H. Wang, “Joint Deep Learning of Facial Expression Synthesis and Recognition,” IEEE Trans Multimedia, vol. 22, no. 11, pp. 2792–2807, Nov. 2020, doi: 10.1109/TMM.2019.2962317.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.
IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
