Facial Performance Sensing Head-Mounted Display

摘要

There are currently no solutions for enabling direct face-to-facerninteraction between virtual reality (VR) users wearing head-mountedrndisplays (HMDs). The main challenge is that the headset obstructs arnsignificant portion of a user’s face, preventing effective facial capturernwith traditional techniques. To advance virtual reality as a nextgenerationrncommunication platform, we develop a novel HMD thatrnenables 3D facial performance-driven animation in real-time. Ourrnwearable system uses ultra-thin flexible electronic materials that arernmounted on the foam liner of the headset to measure surface strainrnsignals corresponding to upper face expressions. These strain signalsrnare combined with a head-mounted RGB-D camera to enhancernthe tracking in the mouth region and to account for inaccuraternHMD placement. To map the input signals to a 3D face model,rnwe perform a single-instance offline training session for each person.rnFor reusable and accurate online operation, we propose a shortrncalibration step to readjust the Gaussian mixture distribution of thernmapping before each use. The resulting animations are visuallyrnon par with cutting-edge depth sensor-driven facial performancerncapture systems and hence, are suitable for social interactions inrnvirtual worlds.