Real-time Shading-based Refinement for Consumer Depth Cameras

摘要

We present the first real-time method for refinement of depth datarnusing shape-from-shading in general uncontrolled scenes. Per frame,rnour real-time algorithm takes raw noisy depth data and an alignedrnRGB image as input, and approximates the time-varying incidentrnlighting, which is then used for geometry refinement. This leads torndramatically enhanced depth maps at 30Hz. Our algorithm makesrnfew scene assumptions, handling arbitrary scene objects even underrnmotion. To enable this type of real-time depth map enhancement,rnwe contribute a new highly parallel algorithm that reformulates therninverse rendering optimization problem in prior work, allowing usrnto estimate lighting and shape in a temporally coherent way at videornframe-rates. Our optimization problem is minimized using a newrnregular grid Gauss-Newton solver implemented fully on the GPU.rnWe demonstrate results showing enhanced depth maps, which arerncomparable to offline methods but are computed orders of magnitudernfaster, as well as baseline comparisons with online filtering-basedrnmethods. We conclude with applications of our higher quality depthrnmaps for improved real-time surface reconstruction and performancerncapture.