High-Fidelity Facial and Speech Animation for VR HMDs

摘要

Significant challenges currently prohibit expressive interaction in virtualrnreality (VR). Occlusions introduced by head-mounted displaysrn(HMDs) make existing facial tracking techniques intractable, andrneven state-of-the-art techniques used for real-time facial tracking inrnunconstrained environments fail to capture subtle details of the user’srnfacial expressions that are essential for compelling speech animation.rnWe introduce a novel system for HMD users to control a digitalrnavatar in real-time while producing plausible speech animation andrnemotional expressions. Using a monocular camera attached to anrnHMD, we record multiple subjects performing various facial expressionsrnand speaking several phonetically-balanced sentences. Thesernimages are used with artist-generated animation data correspondingrnto these sequences to train a convolutional neural network (CNN)rnto regress images of a user’s mouth region to the parameters thatrncontrol a digital avatar. To make training this system more tractable,rnwe use audio-based alignment techniques to map images of multiplernusers making the same utterance to the corresponding animationrnparameters. We demonstrate that this approach is also feasible forrntracking the expressions around the user’s eye region with an internalrninfrared (IR) camera, thereby enabling full facial tracking. Thisrnsystem requires no user-specific calibration, uses easily obtainablernconsumer hardware, and produces high-quality animations of speechrnand emotional expressions. Finally, we demonstrate the quality ofrnour system on a variety of subjects and evaluate its performancernagainst state-of-the-art real-time facial tracking techniques.