Learning to Reconstruct Shape and Spatially-Varying Reflectance from a Single Image

摘要

Reconstructing shape and reflectance properties from images is a highlyunder-constrained problem, and has previously been addressed by usingspecialized hardware to capture calibrated data or by assuming known (orhighly constrained) shape or reflectance. In contrast, we demonstrate thatwe can recover non-Lambertian, spatially-varying BRDFs and complex geometrybelonging to any arbitrary shape class, from a single RGB imagecaptured under a combination of unknown environment illumination andflash lighting. We achieve this by training a deep neural network to regressshape and reflectance from the image. Our network is able to address thisproblem because of three novel contributions: first, we build a large-scaledataset of procedurally generated shapes and real-world complex SVBRDFsthat approximate real world appearance well. Second, single image inverserendering requires reasoning at multiple scales, and we propose a cascadenetwork structure that allows this in a tractable manner. Finally, we incorporatean in-network rendering layer that aids the reconstruction task byhandling global illumination effects that are important for real-world scenes.Together, these contributions allow us to tackle the entire inverse renderingproblem in a holistic manner and produce state-of-the-art results on bothsynthetic and real data.