Video-Audio Driven Real-Time Facial Animation

摘要

We present a real-time facial tracking and animation system basedrnon a Kinect sensor with video and audio input. Our method requiresrnno user-specific training and is robust to occlusions, large head rotations,rnand background noise. Given the color, depth and speechrnaudio frames captured from an actor, our system first reconstructsrn3D facial expressions and 3D mouth shapes from color and depthrninput with a multi-linear model. Concurrently a speaker-independentrnDNN acoustic model is applied to extract phoneme state posteriorrnprobabilities (PSPP) from the audio frames. After that, a lip motionrnregressor refines the 3D mouth shape based on both PSPP andrnexpression weights of the 3D mouth shapes, as well as their confidences.rnFinally, the refined 3D mouth shape is combined with otherrnparts of the 3D face to generate the final result. The whole processrnis fully automatic and executed in real time.rnThe key component of our system is a data-driven regresor forrnmodeling the correlation between speech data and mouth shapes.rnBased on a precaptured database of accurate 3D mouth shapes andrnassociated speech audio from one speaker, the regressor jointly usesrnthe input speech and visual features to refine the mouth shape of arnnew actor. We also present an improved DNN acoustic model. It notrnonly preserves accuracy but also achieves real-time performance.rnOur method efficiently fuses visual and acoustic information for 3Drnfacial performance capture. It generates more accurate 3D mouthrnmotions than other approaches that are based on audio or videorninput only. It also supports video or audio only input for real-time facial animation. We evaluate the performance of our system withrnspeech and facial expressions captured from different actors. Resultsrndemonstrate the efficiency and robustness of our method.