SVBiComm: Sign-Voice Bidirectional Communication System for Normal, “Deaf/Dumb” and Blind People based on Machine Learning

摘要

“deaf/dumb” and blind people face problems in communicating with others with difficulties in dealing with the communication technology. The goal of this paper is to design a desktop human computer interface application that is used to facilitate communication between normal, “deaf/dumb” and blind people. SVBiComm system helps blind person to hear voice saying the word gestured by the” deaf/dumb” while the deaf will receive a gesture representing the word said by the blind. SVBiComm works in two directions, the first direction is processing from video to speech. The animated word gestures are mapped with language knowledge base into text. Then, the relevant audio is generated using Text-to-Speech (TTS) API. The second direction is processing from speech to video. The voice from blind is converted into its corresponding text using Speech-to-Text (STT) API. Then, the natural language is mapped from the database to “deaf/dumb” in a relevant sign language form by using a 3D graphical model. The system was evaluated using a set of 113 sentences with 244 signs. In voice recognition; system recognized words with a percentage of 90% from 19 different persons. For Image recognition the system recognized images with a percentage of 84 % for 21 different persons. SVBiComm system provides many facilities with low cost that could be used in many areas.