Word recognition from Indian Sign Language using Transfer Learning Models and RNN Classifier
Keywords:
ISL, Transfer Learning, RNN classifier, video recognitionAbstract
Indian Sign Language (ISL) plays a crucial role in communication for the hearing-impaired community in India. Developing automated systems for words recognition in ISL can greatly enhance accessibility and inclusivity for individuals with hearing impairments. ISL consists of both static (Images) and dynamic (Videos) words. This paper presents a comprehensive approach to build an ISL recognition system for dynamic words. First, a representative dataset of 210 videos for three color is collected by collaborating with native sign language users, comprising a wide range of hand shapes, movements, and facial expressions. Next, Deep learning architectures, such as VGG19, InceptionV3, DenseNet121 and ResNet50 are used for feature extraction from the video frames to capture the salient characteristics of the hand movements and positions in ISL gestures. These features serve as input representations for classification model. Subsequently, a robust classifier model RNN is used for ISL recognition based on the extracted features. Accuracy of 99%, 97%, 98% and 98% is obtained for model VGG19, InceptionV3, DenseNet121 and ResNet50 respectively. The evaluation is performed on a separate test set, ensuring the generalization capability of the trained classifier. The results obtained highlight the potential of deep learning-based approaches for ISL recognition.
Downloads
References
https://www.who.int/en/news-room/fact-sheets/detail/deafness-and-hearing-loss
Diwakar, S., & Basu, A. (2008). A multilingual multimedia Indian sign language dictionary tool. IJCNLP 2008, 57, 8-13.
Adam, R. (2015). Standardization of sign languages. Sign Language Studies, 15(4), 432-445.
http://censusindia.gov.in/, 2018
Rajam, P. S., & Balakrishnan, G. (2010, July). Indian sign language recognition system to aid deaf-dumb people. In 2010 Second International conference on Computing, Communication and Networking Technologies (pp. 1-9). IEEE.
Sharma, S., & Singh, S. (2022). Recognition of Indian sign language (ISL) using deep learning model. Wireless personal communications, 1-22.
Mahyoub, M., Natalia, F., Sudirman, S., & Mustafina, J. (2023, January). Sign Language Recognition using Deep Learning. In 2023 15th International Conference on Developments in eSystems Engineering (DeSE) (pp. 184-189). IEEE.
El Zaar, A., Benaya, N., & El Allati, A. (2022). Sign language recognition: High performance deep learning approach applyied to multiple sign languages. In E3S Web of Conferences (Vol. 351, p. 01065). EDP Sciences.
Kasukurthi, N., Rokad, B., Bidani, S., & Dennisan, D. A. (2019). American sign language alphabet recognition using deep learning. arXiv preprint arXiv:1905.05487.
Pandian, K., Razman, M. A. M., Khairuddin, I. M., Abdullah, M. A., Ab Nasir, A. F., & Isa, W. H. M. (2023). Sign Language Recognition using Deep Learning through LSTM and CNN. MEKATRONIKA, 5(1), 67-71.
Chu, C., Xiao, Q., Xiao, J., & Gao, C. (2021, September). Sign Language Action Recognition System Based on Deep Learning. In 2021 5th International Conference on Automation, Control and Robots (ICACR) (pp. 24-28). IEEE.
Sharma, S., Gupta, R., & Kumar, A. (2021). Continuous sign language recognition using isolated signs data and deep transfer learning. Journal of Ambient Intelligence and Humanized Computing, 1-12.
Al-Hammadi, M., Muhammad, G., Abdul, W., Alsulaiman, M., Bencherif, M. A., Alrayes, T. S., ... & Mekhtiche, M. A. (2020). Deep learning-based approach for sign language gesture recognition with efficient hand gesture representation. Ieee Access, 8, 192527-192542.
Gupta, R., Golaya, S., & Srinivasan, R. (2022, March). Transfer-Learning Based User-Personalization of Indian Sign Language Recognition System. In 2022 8th International Conference on Advanced Computing and Communication Systems (ICACCS) (Vol. 1, pp. 615-620). IEEE.
Khan, M. D., Patro, B. S., Ranjan, R., Behera, M. C., Kumar, R., & Raj, U. (2021, September). Real-Time American Sign Language Realization Using Transfer Learning With VGG Architecture. In 2021 IEEE 4th International Conference on Computing, Power and Communication Technologies (GUCON) (pp. 1-5). IEEE.
Kishore, P. V. V., & Kumar, P. R. (2012). A video based Indian sign language recognition system (INSLR) using wavelet transform and fuzzy logic. International Journal of Engineering and Technology, 4(5), 537.
Das, S., Biswas, S. K., & Purkayastha, B. (2023, February). Indian Sign Language Recognition System for Emergency Words by Using Shape and Deep Features. In 2023 11th International Conference on Internet of Everything, Microwave Engineering, Communication and Networks (IEMECON) (pp. 1-6). IEEE.
Rokade, Y. I., & Jadav, P. M. (2017). Indian sign language recognition system. International Journal of engineering and Technology, 9(3), 189-196.
Gomathi, V. (2021). Indian Sign Language Recognition through Hybrid ConvNet-LSTM Networks. EMITTER International Journal of Engineering Technology, 9(1), 182-203.
Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2818-2826).
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017). Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4700-4708).
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.
IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
