An Intelligent Multi-Gesture Spotting Robot to Assist Persons with Disabilities

摘要

Sign language is used all over the world by the hearing-impaired and disabled to communicate with each other and the rest of the world. Sign language is more than just moving fingers or hands; it is a viable and visible language in which gestures and facial expressions play a very .important role. These signs can be effectively used as far as communication with humans is concerned but with respect to communication with machines better methodologies and algorithms have to be developed. This paper presents a system which can be used by disabled people (who can communicate only with sign languages) to communicate with machines in order to employ it to their daily life. The system focuses on pose estimation of gestures, used by physically disabled people to give suitable signals to the machines. Pose is generated using silhouettes followed by gesture recognition using Mahalanobis distance metric. The system further was to control a wireless robot from different gestures used by disabled people. The basic American sign language symbols I Hand, II Hand, Aboard, All Gone, etc were recognized and a specific signal with respect to each gesture is transmitted which controls the robot The robot is made to perform, forward, backward, left, right and stop actions for each gesture presented to the system by the disabled .person. The present system can also be modeled to send wireless SMS or emails in worse situations. The system demonstrated potency in the estimation of motion and pose for interpreting the sign language by- using the silhouettes of the pose. The proposed system recognizes sign language, thus providing disabled people a medium to communicate with machines, leading to simplicity in their day to day work.