High-Quality Streamable Free-Viewpoint Video

摘要

We present the first end-to-end solution to create high-quality freeviewpointrnvideo encoded as a compact data stream. Our systemrnrecords performances using a dense set of RGB and IR video cameras,rngenerates dynamic textured surfaces, and compresses these torna streamable 3D video format. Four technical advances contributernto high fidelity and robustness: multimodal multi-view stereo fusingrnRGB, IR, and silhouette information; adaptive meshing guidedrnby automatic detection of perceptually salient areas; mesh trackingrnto create temporally coherent subsequences; and encoding ofrntracked textured meshes as an MPEG video stream. Quantitativernexperiments demonstrate geometric accuracy, texture fidelity, andrnencoding efficiency. We release several datasets with calibratedrninputs and processed results to foster future research.