Displaced Dynamic Expression Regression for Real-time Facial Tracking and Animation

摘要

We present a fully automatic approach to real-time facial trackingrnand animation with a single video camera. Our approach does notrnneed any calibration for each individual user. It learns a genericrnregressor from public image datasets, which can be applied to anyrnuser and arbitrary video cameras to infer accurate 2D facial landmarksrnas well as the 3D facial shape from 2D video frames. Therninferred 2D landmarks are then used to adapt the camera matrixrnand the user identity to better match the facial expressions of therncurrent user. The regression and adaptation are performed in an alternatingrnmanner. With more and more facial expressions observedrnin the video, the whole process converges quickly with accurate facialrntracking and animation. In experiments, our approach demonstratesrna level of robustness and accuracy on par with state-of-theartrntechniques that require a time-consuming calibration step forrneach individual user, while running at 28 fps on average. We considerrnour approach to be an attractive solution for wide deploymentrnin consumer-level applications.