Indian Sign Language Recognition through Hybrid ConvNet-LSTM Networks

  • Muthu Mariappan H Research Scholar
  • Dr Gomathi V Professor and Head of Computer Science and Engineering Department
Keywords: Sign Language Recognition, CasTalk-ISL Dataset, Convolutional Neural Network, Recurrent Neural Network


Dynamic hand gesture recognition is a challenging task of Human-Computer Interaction (HCI) and Computer Vision. The potential application areas of gesture recognition include sign language translation, video gaming, video surveillance, robotics, and gesture-controlled home appliances. In the proposed research, gesture recognition is applied to recognize sign language words from real-time videos. Classifying the actions from video sequences requires both spatial and temporal features. The proposed system handles the former by the Convolutional Neural Network (CNN), which is the core of several computer vision solutions and the latter by the Recurrent Neural Network (RNN), which is more efficient in handling the sequences of movements. Thus, the real-time Indian sign language (ISL) recognition system is developed using the hybrid CNN-RNN architecture. The system is trained with the proposed CasTalk-ISL dataset. The ultimate purpose of the presented research is to deploy a real-time sign language translator to break the hurdles present in the communication between hearing-impaired people and normal people. The developed system achieves 95.99% top-1 accuracy and 99.46% top-3 accuracy on the test dataset. The obtained results outperform the existing approaches using various deep models on different datasets.


Download data is not yet available.


Shweta Dour, Real time recognition of Indian sign language, Ph.D. dissertation, Dept. Elect. Comm. Eng., Bhagwant Univ., Ajmer, Rajasthan, India, 2017.

Zhi-jie Liang, Sheng-bin Liao, Bing-zhang Hu, 3D Convolutional Neural Networks for dynamic sign language recognition, The Computer Journal, vol. 61, no. 11, Nov. 2018, 1724-1736.

P. V. V. Kishore, D. Anil Kumar, A. S. Chandra Sekhara Sastry, E. Kiran Kumar, Motionlets matching with adaptive kernels for 3-D Indian Sign Language Recognition, IEEE Sensors Journal, vol. 18, no. 8, Apr 2018, 3327-3337.

S. Masood, A. Srivastava, H.C. Thuwal, M. Ahmad, Real-time sign language gesture (word) recognition from video sequences using CNN and RNN, Intelligent Engineering Informatics, Advances in Intelligent Systems and Computing, vol. 695, Apr. 2018, 623-632.

Lionel Pigou, Sander Dieleman, Pieter-Jan Kindermans, Benjamin Schrauwen, Sign language recognition using Convolutional Neural Networks, European Conference on Computer Vision- ECCV 2014 Workshops, Lecture Notes in Computer Science, vol. 8925, Mar. 2015, 572-578.

Marwa Elpeltagy, Moataz Abdelwahab, Mohamed E. Hussein, Amin Shoukry, Asmaa Shoala, Moustafa Galal, Multi-modality-based Arabic Sign Language recognition, IET Computer Vision, vol. 12, no. 7, Oct 2018, 1031-1039.

Biplab Ketan Chakraborty, DebajitSarma, M.K. Bhuyan, Karl F MacDorman, Review of constraints on vision-based gesture recognition for human–computer interaction, IET Computer Vision, vol. 12, no. 1, Jan. 2018, 3-15.

Sushmita Mitra, Tinku Acharya, Gesture recognition: A survey, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), vol. 37, no. 3, Apr. 2007, 311-324.

Kouichi Murakami, Hitomi Taguchi, Gesture recognition using Recurrent Neural Networks, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Louisiana, USA, May. 1991, 237-242.

Kenneth Lai, Svetlana N. Yanushkevich, CNN+RNN depth and skeleton based dynamic hand gesture recognition, 24th International Conference on Pattern Recognition (ICPR), Beijing, China, Aug. 2018.

Juan C. Nunez, Raul Cabido, Juan J. Pantrigo, Antonio S. Montemayor, Jose F. Velez, Convolutional Neural Networks and Long Short-Term Memory for skeleton-based human activity and hand gesture recognition, Pattern Recognition, vol. 76, Apr. 2018, 80-94.

Jie Huang, Wengang Zhou, Qilin Zhang, Houqiang Li and Weiping Li, Video-based Sign Language Recognition without Temporal Segmentation, AAAI Conference on Artificial Intelligence (AAAI), 2018.

Q. De Smedt, H. Wannous, J.P. Vandeborre, Skeleton-based dynamic hand gesture recognition, The IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Jun. 2016, 1206–1214.

Hanjie Wang, Xiujuan Chai, Xiaopeng Hong, Guoying Zhao, and Xilin Chen, Isolated sign language recognition with Grassmann covariance matrices, ACM Transactions on Accessible Computing (TACCESS), vol. 8, no. 4, May 2016.

Ronchetti, F., Quiroga, F., Estrebou, C.A., Lanzarini, L.C., Rosete, A, LSA64: an Argentinian sign language dataset, XXII Congreso Argentino de Ciencias de la Computación (CACIC), (2016).

Zafar Ahmed Ansari, Gaurav Harit, Nearest neighbour classification of Indian sign language gestures using kinect camera, Sadhana, 41, Feb. 2016, 161 – 182.

Escalera, S., Bar, X., Gonzlez, J., Bautista, M.A., Madadi, M., Reyes, M., Ponce, V., Escalante, H.J., Shotton, J., Guyon, Cha, Learn Looking at People Challenge 2014: Dataset and Results, Computer Vision - ECCV 2014 Workshops, Lecture Notes in Computer Science, vol 8925.

S. Escalera, J. Gonzàlez, X. Baró, M. Reyes, O. Lopes, I. Guyon, V. Athistos, H.J. Escalante, Multi-modal Gesture Recognition Challenge 2013: Dataset and Results, ICMI 2013.

Shugao Ma, Sarah Adel Bargal, Jianming Zhang, Leonid Sigal, Stan Sclaroff, Do less and achieve more: Training CNNs for action recognition utilizing action images from the Web, Pattern Recognition, vol. 68, Aug 2017, 334-345.

Julien Maitre, Clement Rendu, Kevin Bouchard, Bruno Bouchard, Sebastien Gaboury, Basic daily activity recognition with a data glove, The 10th International Conference on Ambient Systems, Networks and Technologies (ANT), vol. 151, May 2019, 108-115.

Wen-Ren Yang, Chau-Shing Wang, Chien-Pu Chen, Motion-pattern recognition system using a wavelet-neural network, IEEE Transactions on Consumer Electronics, vol. 65, no. 2, May 2019, 170-178.

Danilo Avola, Marco Bernardi, Luigi Cinque, Gian Luca Foresti, Cristiano Massaroni, Exploiting Recurrent Neural Networks and leap motion controller for the recognition of sign language and semaphoric hand gestures, IEEE Transactions on Multimedia, vol. 21, no. 1, Jan. 2019, 234-245.

Bo Li, Chao Zhang, Cheng Han, Baoxing Bai, Gesture Recognition Based on Kinect v2 and Leap Motion Data Fusion, International Journal of Pattern Recognition and Artificial Intelligence, vol. 33, no. 05, 1955005 (2019).

Shuiwang Ji, Wei Xu, Ming Yang, Member, Kai Yu, 3D Convolutional Neural Networks for human action recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 1, Jan. 2013.

Kensho Hara, Hirokatsu Kataoka, Yutaka Satoh, Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition, Aug 2017. [].

Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun, Faster R-CNN: Towards real-time object detection with region proposal networks, Jan. 2016. [].

Karen Simonyan, Andrew Zisserman, Very deep Convolutional Networks for large-scale image recognition, ICLR 2015, Apr. 2015.

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, Zbigniew Wojna, Rethinking the Inception architecture for computer vision, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, Dec. 2016.

Ilias Papastratis, Kosmas Dimitropoulos, Dimitrios Konstantinidis, Petros Daras, Continuous Sign Language Recognition Through Cross-Modal Alignment of Video and Text Embeddings in a Joint-Latent Space, IEEE Access, Vol. 8, May 2020, 91170-91180.

Muneer Al-Hammadi, Ghulam Muhammad, Wadood Abdul, Mansour Alsulaiman, Mohamed A. Bencherif, Mohamed Amine Mekhtiche, Hand Gesture Recognition for Sign Language Using 3DCNN, IEEE Access, Vol. 8, May 2020, 79491-79509.

Safa Ameur, Anouar Ben Khalifa, Med Salim Bouhlel, A novel hybrid bidirectional unidirectional LSTM network for dynamic hand gesture recognition with Leap Motion, Entertainment Computing, Vol 35, June 2020.

Elahe Rahimian, Soheil Zabihi, Seyed Farokh Atashzar, Amir Asif, Arash Mohammadi, Surface EMG-Based Hand Gesture Recognition via Hybrid and Dilated Deep Neural Network Architectures for Neurorobotic Prostheses, Journal of Medical Robotics Research, Vol. 5, No. 1, Mar. 2020.

Sepp Hochreiter, Jurgen Schmidhuber, Long Short-Term Memory, Neural Computation, vol. 9, no. 8, Nov. 1997, 1735-1780.

Zhigang Tu, Hongyan Li, Dejun Zhang, Justin Dauwels, Baoxin Li, Junsong Yuan, Action-stage emphasized spatiotemporal VLAD for video action recognition, IEEE Transactions on Image Processing, vol. 28, no. 6, Jun. 2019, 2799-2812.

Caihua Liu, Jie Liu, Zhicheng He, YujiaZhai, Qinghua Hu, Yalou Huang, Convolutional neural random fields for action recognition, Pattern Recognition, vol. 59, Nov. 2016, 213-224.

PichaoWang, Wanqing Li, Philip Ogunbona, Jun Wan, Sergio Escalera, RGB-D-based human motion recognition with deep learning: A survey, Computer Vision and Image Understanding, vol. 171, Jun. 2018, 118-139.

Santanu Pattanayak, Foundations of artificial intelligence-based systems, in Intelligent Projects using Python: 9 real-world AI projects leveraging machine learning and deep learning with TensorFlow and Keras, Packt Publishing, Birmingham, UK, 2019, Ch. 1. Sec. 13-14. [

Festus Osayamwen, Jules-Raymond Tapamo, Deep learning class discrimination based on prior probability for human activity recognition, IEEE Access, vol. 7, Feb. 2019, 14747 - 14756.

Di Wu, Lionel Pigou, Pieter-Jan Kindermans, Nam Do-Hoang Le, Ling Shao, Joni Dambre, Jean-Marc Odobez, Deep Dynamic Neural Networks for multimodal gesture segmentation and recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 8, Aug. 2016, 1583-1597.

Runpeng Cui, Hu Liu, Changshui Zhang, A deep neural framework for continuous sign language recognition by iterative training, IEEE Transactions on Multimedia, vol. 21, no. 7, Jul. 2019, 1880-1891.

Fangxin Wang, Wei Gong, Jiangchuan Liu, On spatial diversity in wiFi-based human activity recognition: A deep learning-based approach, IEEE Internet of Things Journal, vol. 6, no. 2, Apr. 2019, 2035-2047.

Yanqiu Liao, PengwenXiong, WeidongMin, Weiqiong Min, Jiahao Lu, Dynamic sign language recognition based on video sequence with BLSTM-3D residual networks, IEEE Access, vol. 7, Mar. 2019, 38044 – 38054.

Hao Tang, Hong Liu, Wei Xiao, Nicu Sebe, Fast and robust dynamic hand gesture recognition via key frames extraction and feature fusion, Neurocomputing, vol.331, Feb. 2019, 424-433.

Google Colaboratory []

How to Cite
Muthu Mariappan H, & Dr Gomathi V. (2021). Indian Sign Language Recognition through Hybrid ConvNet-LSTM Networks. EMITTER International Journal of Engineering Technology, 9(1), 182-203.

Funding data