Real-Time High-Fidelity Facial Performance Capture

摘要

We present the first real-time high-fidelity facial capture method.rnThe core idea is to enhance a global real-time face tracker, whichrnprovides a low-resolution face mesh, with local regressors that addrnin medium-scale details, such as expression wrinkles. Our main observationrnis that although wrinkles appear in different scales and atrndifferent locations on the face, they are locally very self-similar andrntheir visual appearance is a direct consequence of their local shape.rnWe therefore train local regressors from high-resolution capturerndata in order to predict the local geometry from local appearancernat runtime. We propose an automatic way to detect and align the localrnpatches required to train the regressors and run them efficientlyrnin real-time. Our formulation is particularly designed to enhancernthe low-resolution global tracker with exactly the missing expressionrnfrequencies, avoiding superimposing spatial frequencies in thernresult. Our system is generic and can be applied to any real-timerntracker that uses a global prior, e.g. blend-shapes. Once trained,rnour online capture approach can be applied to any new user withoutrnadditional training, resulting in high-fidelity facial performance reconstructionrnwith person-specific wrinkle details from a monocularrnvideo camera in real-time.