3D Realistic Talking Face Co-Driven by Text and Speech

摘要

To create 3D realistic talking face has been a challenge for a long time. Previous works emphasize text or speech driven talking face respectively while the animation result is not very realistic or natural- looking. In the proposed approach, text and speech are considered to drive the 3D talkingface coordinately. The text is translated into a sequence of visemes' transcription. And time vector of the sequence is extracted from the speech corresponding to the text after it is segmented into phonetic sequence. A muscle based viseme vector is defined for static viseme. And then, with the time vector and the static visemes's sequence, dynamic visemes are generated through time-related dominance function. Finally, according to the frame rate to be rendered, intermediate frames are interpolated between key frames to make the animation result looks more natural and realistic than those obtained based on the text or speech-driven only.