The Relightables: Volumetric Performance Capture of Humans with Realistic Relighting

摘要

We present “The Relightables”, a volumetric capture system for photorealisticand high quality relightable full-body performance capture. While significantprogress has been made on volumetric capture systems, focusing on 3Dgeometric reconstruction with high resolution textures, much less workhas been done to recover photometric properties needed for relighting.Results from such systems lack high-frequency details and the subject’sshading is prebaked into the texture. In contrast, a large body of workhas addressed relightable acquisition for image-based approaches, whichphotograph the subject under a set of basis lighting conditions and recombinethe images to show the subject as they would appear in a target lightingenvironment. However, to date, these approaches have not been adaptedfor use in the context of a high-resolution volumetric capture system. Ourmethod combines this ability to realistically relight humans for arbitraryenvironments, with the benefits of free-viewpoint volumetric capture andnew levels of geometric accuracy for dynamic performances. Our subjectsare recorded inside a custom geodesic sphere outfitted with 331 custom colorLED lights, an array of high-resolution cameras, and a set of custom highresolutiondepth sensors. Our system innovates in multiple areas: First, wedesigned a novel active depth sensor to capture 12.4MP depth maps, whichwe describe in detail. Second, we show how to design a hybrid geometric andmachine learning reconstruction pipeline to process the high resolution inputand output a volumetric video. Third, we generate temporally consistentreflectance maps for dynamic performers by leveraging the information contained in two alternating color gradient illumination images acquiredat 60 Hz. Multiple experiments, comparisons, and applications show thatThe Relightables significantly improves upon the level of realism in placingvolumetrically captured human performances into arbitrary CG scenes.