Aggregation Signature of Multi Scale Features from Super Resolution Images for Bharatanatyam Mudra Classification for Augmented Reality Based Learning

Saba  Naaz; K. B.  ShivaKumar; Parameshachari  B. D.

Authors

Saba Naaz Visvesvaraya Technological University. Belagavi, Karnataka
K. B. ShivaKumar HOD, Dept of CSE, Don Bosco Institute Technology, Bengaluru
Parameshachari B. D. Professor, Dept of E&CE, NITTE Meenakshi Institute of Technology, Bengaluru.

Keywords:

communication, orientation, resolution, Aggregation, classification

Abstract

Hand gesture is an important non verbal communication mechanism of Indian classical dances especially Bharatanatyam. The hand gestures in Bharatanatyam are called as mudras and there are total 52 mudras with 28 single hand mudras and 24 double hand mudras. Many computer aid mudras classification systems were designed to infer the non verbal theme communicated via mudras. But unlike other hand gesture recognition system, accurate classification of mudra is challenging due to high structural similarity between mudras. This work proposesdeep learning multi scale feature guided aggregated signature for accurate classification of mudras. The deep learning multi scalefeatures are extracted from multi scale images after super resolution and thus self similarities between mudras can be easily differentiated. In addition the features are scale and orientation invariant. Aggregation signature is constructed based on multi scale super resolution features to reduce the classification time. The proposed solution is able to provide an average accuracy of 96% which is atleast 2% higher compared to existing works. Finally the proof of concept of application of proposed mudra classification system in augmented reality based learning system is presented.

Downloads

Download data is not yet available.

References

Mallik, A., Chaudhury, S., and Ghosh, H. Nrityakosha: Preserving the Intangible Heritage of Indian Classical Dance. Journal on Computing and Cultural Heritage (JOCCH) 4, 3 (2011), 11

Tanwi Mallick, Partha Pratim Das, Arun Kumar Majumdar:Bharatanatyam Dance Transcription using Multimedia Ontology and Machine Learning. CoRR abs/2004.11994 (2020)

Sangeeta Jadav, and Sasikumar Mukundan, “A Computational Model for Bharata Natyam Choreography,” (IJCSIS) International Journal of Computer Science and Information Security, Vol. 8, No. 7, October, 2010.

Anami, B.S., & Bhandage, V.A. (2018). A Comparative Study of Suitability of Certain Features in Classification of Bharatanatyam Mudra Images Using Artificial Neural Network. Neural Processing Letters, 50, 741-769.

Saha, S., Ghosh, L., Konar, A. and Janarthanan, R. (2013) ‘Fuzzy L membership function based hand gesture recognition for Bharatanatyam dance’, Proceedings of the 5th International conference on Computational Intelligence and Communication Networks (CICN), pp.331–335, IEEE

A. P. Parameshwaran, H. P. Desai, R. Sunderraman and M. Weeks, "Transfer Learning for Classifying Single Hand Gestures on Comprehensive Bharatanatyam Mudra Dataset," 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2019, pp. 508-510

B.S. Anami and V.A. Bhandage, "A vertical-horizontal-intersections feature based method for identification of bharatanatyam double hand mudra images", Multimedia Tools and Applications, vol. 77, no. 23, pp. 31021-31040, 2018

K.V.V. Kumar and P.V.V. Kishore, "Indian Classical Dance Mudra Classification Using HOG Features and SVM Classifier", International Journal of Electrical & Computer Engineering (2088–8708), vol. 7, no. 5, 2017

Kopuklu, Okan & Gunduz, Ahmet & Kose, Neslihan & Rigoll, Gerhard. (2019). Real-time Hand Gesture Detection and Classification Using Convolutional Neural Networks. 1-8. 10.1109/FG.2019.8756576.

Sahoo JP, Prakash AJ, Pławiak P, Samantray S. Real-Time Hand Gesture Recognition Using Fine-Tuned Convolutional Neural Network. Sensors. 2022; 22(3):706.

Patil, A.R.; Subbaraman, S. A spatiotemporal approach for vision-based hand gesture recognition using Hough transform and neural network. Signal, Image Video Process. 2019, 13, 413–421

Fang, L.; Liang, N.; Kang, W.; Wang, Z.; Feng, D.D. Real-time hand posture recognition using hand geometric features and fisher vector. Signal Process. Image Commun. 2020, 82, 115729

Gadekallu, Thippa & Srivastava, Gautam & Liyanage, Madhusanka & Meenakshisundaram, Iyapparaja & Chowdhary, Chiranji & Koppu, Srinivas & Reddy, Praveen. (2022). Hand gesture recognition based on a Harris Hawks optimized Convolution Neural Network. Computers & Electrical Engineering. 100. 107836. 10.1016/j.compeleceng.2022.107836.

Lim, Kian & Tan, Alan & Tan, Shing. (2016). A Feature Covariance Matrix with Serial Particle Filter for Isolated Sign Language Recognition. Expert Systems with Applications. 54. 10.1016/j.eswa.2016.01.047.

Jain N, Bansal V, Virmani D, Gupta V, Salas-Morera L, Garcia-Hernandez L. An Enhanced Deep Convolutional Neural Network for Classifying Indian Classical Dance Forms. Applied Sciences. 2021; 11(14):6253.

Kumar, K.V.V.; Kishore, P.V.V.; Kumar, D.A. Indian Classical Dance Classification with Adaboost Multiclass Classifier on Multifeature Fusion. Math. Probl. Eng. 2017, 2017, 1–18.

Kishore, P.V.V.; Kumar, K.V.V.; Kumar, E.K.; Sastry, A.S.C.S.; Kiran, M.T.; Kumar, D.A.; Prasad, M.V.D. Indian Classical Dance Action Identification and Classification with Convolutional Neural Networks. Adv. Multimedia 2018, 2018, 1–10

S. Mozarkar and C. S. Warnekar, “Recognizing bharatnatyam mudra using principles of gesture recognition gesture recognition,” International Journal of Computer Science and Network, vol. 2, no. 2, pp. 46–52, 2013

Kishore, P.V.V., Kishore, S.R.C., Prasad, M.V.D.: Conglomeration of hand shapes and texture information for recognizing gestures of Indian sign language using feed forward neural networks. Int. J. Eng. Technol. (IJET), ISSN: 0975-4024 (2013).

https://www.kaggle.com/datasets/ishanishah8/indian-classical-mudras-classification

https://www.kaggle.com/competitions/GestureChallenge/overview/data-description-2