MDS: A Multimodal-based Dialog System

摘要

This paper describes MDS: a Multimodeal-based Dialog System that supports communication between the hearing imparied and hearing-abled. The system converts sign language to speech, and combines speech with gesture and lip motion using a human face. The features of the human face are derived by doing a 3D feature extraction f the speacker's face, so that the 'virtual face" similar to the actural specaker. The main technologies associated wit hthe system include sign language recognition, sign language synthesis and the synchrony of the lip movement and speech. Integration of the sign language recognition, sign language synthesis and the synchrony of the lip movement and speech. Integration of the sign language recognition, sign language synthesis speech recognition, speech synthesis and 3D virtual human technologies provides a new way to interact with computers for the hearing impaired.