Novel Emotion Recognition Framework from Facial Expressions Using Spiking Neural Networks on Wearable Edge Devices

Authors

  • A.Jasvant Ram, K. Prema, N. Susila, K. Jaya Deepthi, C. Sakthi Lakshmi Priya , S. Manikandan

Keywords:

Spiking Neural Networks (SNNs), Open Neural Network Exchange (ONNX), Long Short-Term Memory (LSTM), Message Queuing Telemetry Transport (MQTT).

Abstract

Real-time emotion recognition from facial expressions holds significant promise for enhancing human-computer interaction and personalizing user experiences. Harnessing the potential of advanced technologies, the research presents a novel approach to real-time emotion recognition from facial expressions using Spiking Neural Networks (SNN) on wearable edge devices. The methodology integrates key technologies including Open Neural Network Exchange (ONNX), Message Queuing Telemetry Transport (MQTT), and Long Short-Term Memory (LSTM) networks to enhance the efficiency and accuracy of emotion recognition systems in practical scenarios. By leveraging ONNX, seamless model interchangeability and deployment across diverse hardware platforms are achieved, ensuring scalability and flexibility in model deployment. Through optimized model conversion and deployment on wearable edge devices, interoperability and efficiency in real-time emotion recognition tasks are ensured. MQTT serves as a lightweight and reliable communication protocol for seamless data exchange between wearable edge devices and external systems, facilitating real-time transmission of facial expression data and inference results. This enables collaborative processing and decision-making across distributed networks, enhancing system responsiveness and scalability. The integration of LSTM networks captures temporal dependencies in facial expressions, improving the accuracy and robustness of emotion recognition systems. LSTM networks excel in modeling sequential data and long-term dependencies, making them suitable for analyzing temporal patterns in facial expressions over time.Top of Form

Downloads

Download data is not yet available.

References

Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001). Emotion recognition in human-computer interaction. IEEE Signal processing magazine, 18(1), 32-80.

Zhang, S., Li, Y., Zhang, S., Shahabi, F., Xia, S., Deng, Y., & Alshurafa, N. (2022). Deep learning in human activity recognition with wearable sensors: A review on advances. Sensors, 22(4), 1476.

Shi, Q., Dong, B., He, T., Sun, Z., Zhu, J., Zhang, Z., & Lee, C. (2020). Progress in wearable electronics/photonics—Moving toward the era of artificial intelligence and internet of things. InfoMat, 2(6), 1131-1162.

Seng, K. P., Lee, P. J., & Ang, L. M. (2021). Embedded intelligence on FPGA: Survey, applications and challenges. Electronics, 10(8), 895.

Mazumder, A. N., Meng, J., Rashid, H. A., Kallakuri, U., Zhang, X., Seo, J. S., & Mohsenin, T. (2021). A survey on the optimization of neural network accelerators for micro-ai on-device inference. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 11(4), 532-547.

Azghadi, M. R., Lammie, C., Eshraghian, J. K., Payvand, M., Donati, E., Linares-Barranco, B., & Indiveri, G. (2020). Hardware implementation of deep network accelerators towards healthcare and biomedical applications. IEEE Transactions on Biomedical Circuits and Systems, 14(6), 1138-1159.

Ferguson, M., Jeong, S., Law, K. H., Levitan, S., Narayanan, A., Burkhardt, R., ... & Lee, Y. T. T. (2019, August). A standardized representation of convolutional neural networks for reliable deployment of machine learning models in the manufacturing industry. In International Design Engineering Technical Conferences and Computers and Information in Engineering Conference (Vol. 59179, p. V001T02A005). American Society of Mechanical Engineers.

Tosetti, F. (2021). Deep Learning on the Edge: a comparative analysis on Computer Vision for space applications (Doctoral dissertation, Politecnico di Torino).

Dizdarević, J., Carpio, F., Jukan, A., & Masip-Bruin, X. (2019). A survey of communication protocols for internet of things and related challenges of fog and cloud computing integration. ACM Computing Surveys (CSUR), 51(6), 1-29.

Koziolek, H., Grüner, S., & Rückert, J. (2020). A comparison of MQTT brokers for distributed IoT edge computing. In Software Architecture: 14th European Conference, ECSA 2020, L'Aquila, Italy, September 14–18, 2020, Proceedings 14 (pp. 352-368). Springer International Publishing.

Teixeira, T., Granger, É., & Lameiras Koerich, A. (2021). Continuous emotion recognition with spatiotemporal convolutional neural networks. Applied Sciences, 11(24), 11738.

Zheng, Y., & Blasch, E. (2023). Facial Micro-Expression Recognition Enhanced by Score Fusion and a Hybrid Model from Convolutional LSTM and Vision Transformer. Sensors, 23(12), 5650.

Geetha, A. V., Mala, T., Priyanka, D., & Uma, E. (2024). Multimodal Emotion Recognition with deep learning: advancements, challenges, and future directions. Information Fusion, 105, 102218.

Rathi, N., Agrawal, A., Lee, C., Kosta, A. K., & Roy, K. (2021, February). Exploring spike-based learning for neuromorphic computing: Prospects and perspectives. In 2021 Design, Automation & Test in Europe Conference & Exhibition (DATE) (pp. 902-907). IEEE.

Kaushik, H., Kumar, T., & Bhalla, K. (2022). iSecureHome: A deep fusion framework for surveillance of smart homes using real-time emotion recognition. Applied Soft Computing, 122, 108788.

Datta, G. (2023). Towards Efficient Edge Intelligence with In-Sensor and Neuromorphic Computing: Algorithm-Hardware Co-Design (Doctoral dissertation, University of Southern California).

Pu, G., & Chen, J. (2024). Facial Expression Recognition Based on Convolutional Spiking Neural Network and STDP Fine-Tune.

Aouedi, O. (2024). Towards a Scalable and Energy-Efficient Framework for Industrial Cloud-Edge-IoT Continuum. IEEE Internet of Things Magazine.

Tan, C., Ceballos, G., Kasabov, N., & Puthanmadam Subramaniyam, N. (2020). Fusionsense: Emotion classification using feature fusion of multimodal data and deep learning in a brain-inspired spiking neural network. Sensors, 20(18), 5328.

Shuvo, M. M. H., Islam, S. K., Cheng, J., & Morshed, B. I. (2022). Efficient acceleration of deep learning inference on resource-constrained edge devices: A review. Proceedings of the IEEE, 111(1), 42-91.

Hassija, V., Chamola, V., Mahapatra, A., Singal, A., Goel, D., Huang, K., ... & Hussain, A. (2024). Interpreting black-box models: a review on explainable artificial intelligence. Cognitive Computation, 16(1), 45-74.

Downloads

Published

06.08.2024

How to Cite

A.Jasvant Ram. (2024). Novel Emotion Recognition Framework from Facial Expressions Using Spiking Neural Networks on Wearable Edge Devices. International Journal of Intelligent Systems and Applications in Engineering, 12(23s), 51–61. Retrieved from https://ijisae.org/index.php/IJISAE/article/view/6432

Issue

Section

Research Article