LookinGood: Enhancing Performance Capture with Real-time Neural Re-Rendering

摘要

Motivated by augmented and virtual reality applications such as telepresence,there has been a recent focus in real-time performance capture of humansunder motion. However, given the real-time constraint, these systems oftensuffer from artifacts in geometry and texture such as holes and noise inthe final rendering, poor lighting, and low-resolution textures. We take thenovel approach to augment such real-time performance capture systemswith a deep architecture that takes a rendering from an arbitrary viewpoint,and jointly performs completion, super resolution, and denoising of theimagery in real-time. We call this approach neural (re-)rendering, and our live system “LookinGood". Our deep architecture is trained to produce highresolution and high quality images from a coarse rendering in real-time. First,we propose a self-supervised training method that does not require manualground-truth annotation. We contribute a specialized reconstruction errorthat uses semantic information to focus on relevant parts of the subject, e.g.the face. We also introduce a salient reweighing scheme of the loss functionthat is able to discard outliers. We specifically design the system for virtualand augmented reality headsets where the consistency between the left andright eye plays a crucial role in the final user experience. Finally, we generatetemporally stable results by explicitly minimizing the difference between twoconsecutive frames.We tested the proposed system in two different scenarios:one involving a single RGB-D sensor, and upper body reconstruction of anactor, the second consisting of full body 360◦ capture. Through extensiveexperimentation, we demonstrate how our system generalizes across unseensequences and subjects.