Radiative Backpropagation: An Adjoint Method for Lightning-Fast Differentiable Rendering

摘要

Physically based diffierentiable rendering has recently evolved into a powerfultool for solving inverse problems involving light. Methods in this area performa diffierentiable simulation of the physical process of light transport andscattering to estimate partial derivatives relating scene parameters to pixelsin the rendered image. Together with gradient-based optimization, such algorithmshave interesting applications in diverse disciplines, e.g., to improvethe reconstruction of 3D scenes, while accounting for interreection andtransparency, or to design meta-materials with specified optical properties.The most versatile diffierentiable rendering algorithms rely on reversemodediffierentiation to compute all requested derivatives at once, enablingoptimization of scene descriptions with millions of free parameters. However,a severe limitation of the reverse-mode approach is that it requiresa detailed transcript of the computation that is subsequently replayed toback-propagate derivatives to the scene parameters. The transcript of typicalrenderings is extremely large, exceeding the available system memory bymany orders of magnitude, hence current methods are limited to simplescenes rendered at low resolutions and sample counts.We introduce radiative backpropagation, a fundamentally diffierent approachto diffierentiable rendering that does not require a transcript, greatlyimproving its scalability and eciency. Our main insight is that reversemodepropagation through a rendering algorithm can be interpreted as thesolution of a continuous transport problem involving the partial derivative ofradiance with respect to the optimization objective. This quantity is “emittedâ€by sensors, “scattered†by the scene, and eventually “received†by objectswith diffierentiable parameters. Diffierentiable rendering then decomposesinto two separate primal and adjoint simulation steps that scale to complexscenes rendered at high resolutions. We also investigated biased variantsof this algorithm and find that they considerably improve both runtimeand convergence speed. We showcase an ecient GPU implementation ofradiative backpropagation and compare its performance and the quality ofits gradients to prior work.