Mudra: Convolutional Neural Network based Indian Sign Language Translator for Banks

摘要

Signlanguageisamediumofexpressing thoughts and feelings by the deaf-dumb community. It could be extremely challenging for deaf-mute people to communicate efficiently in banks, where they might have to explain their needs. There are very few people who can understand sign language. The main focus of our proposed method is to design an ISL (Indian Sign Language) hand gesture motion translation tool for banks for helping the deaf-mute community to convey their ideas by converting them to text format. In the fields of ASL (American Sign Language) and other languages, ample amounts of work have been done. Apart from other algorithms, our proposed method recognizes human actions considering isolated dynamic Indian signs related to the bank as a novel approach. There are very few research works carried out in this field of ISL recognition for banks. Over and above that, an insufficient amount of dataset along with dissimilarity in gestures length was a difficulty. We used a self-recorded ISL dataset for training the model for recognizing the gestures. Unlike image data, the video domain was a new challenge. Larger lengthened video gestures were taken and actions were recognized from a series of video frames. CNN (Convolutional Neural Network) named inception V3 was used to extract the image features. LSTM (Long Short Term Memory), an architecture of RNN (Recurrent neural network) classified these gestures and are translated into text. Experimental results display that this approach towards isolated word dynamic hand gesture recognition systems provides an accurate and effective method for the interaction between non-signer and signer.