Corrective 3D Reconstruction of Lips from Monocular Video

摘要

In facial animation, the accurate shape and motion of the lips ofrnvirtual humans is of paramount importance, since subtle nuances inrnmouth expression strongly influence the interpretation of speech andrnthe conveyed emotion. Unfortunately, passive photometric reconstructionrnof expressive lip motions, such as a kiss or rolling lips, isrnfundamentally hard even with multi-view methods in controlled studios.rnTo alleviate this problem, we present a novel approach for fullyrnautomatic reconstruction of detailed and expressive lip shapes alongrnwith the dense geometry of the entire face, from just monocularrnRGB video. To this end, we learn the difference between inaccuraternlip shapes found by a state-of-the-art monocular facial performancerncapture approach, and the true 3D lip shapes reconstructed usingrna high-quality multi-view system in combination with applied liprntattoos that are easy to track. A robust gradient domain regressorrnis trained to infer accurate lip shapes from coarse monocular reconstructions,rnwith the additional help of automatically extracted innerrnand outer 2D lip contours. We quantitatively and qualitatively showrnthat our monocular approach reconstructs higher quality lip shapes,rneven for complex shapes like a kiss or lip rolling, than previousrnmonocular approaches. Furthermore, we compare the performancernof person-specific and multi-person generic regression strategies andrnshow that our approach generalizes to new individuals and generalrnscenes, enabling high-fidelity reconstruction even from commodityrnvideo footage.