首页>
外国专利>
PERSONALIZED SPEECH-TO-VIDEO WITH THREE-DIMENSIONAL (3D) SKELETON REGULARIZATION AND EXPRESSIVE BODY POSES
PERSONALIZED SPEECH-TO-VIDEO WITH THREE-DIMENSIONAL (3D) SKELETON REGULARIZATION AND EXPRESSIVE BODY POSES
展开▼
机译:具有三维(3D)骨架正规化和富有表现力的身体的个性化语音 - 视频
展开▼
页面导航
摘要
著录项
相似文献
摘要
Presented herein are novel embodiments for converting a given speech audio or text into a photo-realistic speaking video of a person with synchronized, realistic, and expressive body dynamics. In one or more embodiments, 3D skeleton movements are generated from the audio sequence using a recurrent neural network, and an output video is synthesized via a conditional generative adversarial network. To make movements realistic and expressive, the knowledge of an articulated 3D human skeleton and a learned dictionary of personal speech iconic gestures may be embedded into the generation process in both learning and testing pipelines. The former prevents the generation of unreasonable body distortion, while the later helps the model quickly learn meaningful body movement with a few videos. To produce photo-realistic and high-resolution video with motion details, a part-attention mechanism is inserted in the conditional GAN, where each detailed part is automatically zoomed in to have their own discriminators.
展开▼