Real-time Non-rigid Reconstruction using an RGB-D Camera

摘要

We present a combined hardware and software solution for markerlessrnreconstruction of non-rigidly deforming physical objects withrnarbitrary shape in real-time. Our system uses a single self-containedrnstereo camera unit built from off-the-shelf components and consumerrngraphics hardware to generate spatio-temporally coherentrn3D models at 30 Hz. A new stereo matching algorithm estimatesrnreal-time RGB-D data. We start by scanning a smooth templaternmodel of the subject as they move rigidly. This geometric surfacernprior avoids strong scene assumptions, such as a kinematic humanrnskeleton or a parametric shape model. Next, a novel GPU pipelinernperforms non-rigid registration of live RGB-D data to the smoothrntemplate using an extended non-linear as-rigid-as-possible (ARAP)rnframework. High-frequency details are fused onto the final mesh usingrna linear deformation model. The system is an order of magnitudernfaster than state-of-the-art methods, while matching the quality andrnrobustness of many offline algorithms. We show precise real-timernreconstructions of diverse scenes, including: large deformations ofrnusers’ heads, hands, and upper bodies; fine-scale wrinkles and foldsrnof skin and clothing; and non-rigid interactions performed by usersrnon flexible objects such as toys. We demonstrate how acquired modelsrncan be used for many interactive scenarios, including re-texturing,rnonline performance capture and preview, and real-time shape andrnmotion re-targeting.