Deep Blending for Free-Viewpoint Image-Based Rendering

摘要

Free-viewpoint image-based rendering (IBR) is a standing challenge. IBRmethods combine warped versions of input photos to synthesize a novelview. The image quality of this combination is directly afected by geometricinaccuracies of multi-view stereo (MVS) reconstruction and by view- andimage-dependent efects that produce artifacts when contributions from differentinput views are blended. We present a new deep learning approach toblending for IBR, in which we use held-out real image data to learn blendingweights to combine input photo contributions. Our Deep Blending methodrequires us to address several challenges to achieve our goal of interactivefree-viewpoint IBR navigation.We irst need to provide suiciently accurategeometry so the Convolutional Neural Network (CNN) can succeed in indingcorrect blending weights. We do this by combining two diferent MVSreconstructions with complementary accuracy vs. completeness tradeofs.To tightly integrate learning in an interactive IBR system, we need to adaptour rendering algorithm to produce a ixed number of input layers thatcan then be blended by the CNN. We generate training data with a varietyof captured scenes, using each input photo as ground truth in a held-outapproach. We also design the network architecture and the training loss toprovide high quality novel view synthesis, while reducing temporal lickeringartifacts. Our results demonstrate free-viewpoint IBR in a wide varietyof scenes, clearly surpassing previous methods in visual quality, especiallywhen moving far from the input cameras.