Coherent multi-dimensional segmentation of multiview images using a variational framework and applications to image based rendering

摘要

Image Based Rendering (IBR) and in particular light field rendering has attracted a lot ofudattention for interpolating new viewpoints from a set of multiview images. New images ofuda scene are interpolated directly from nearby available ones, thus enabling a photorealisticudrendering. Sampling theory for light fields has shown that exact geometric informationudin the scene is often unnecessary for rendering new views. Indeed, the band of the functionudis approximately limited and new views can be rendered using classical interpolationudmethods. However, IBR using undersampled light fields suffers from aliasing effects andudis difficult particularly when the scene has large depth variations and occlusions. In orderudto deal with these cases, we study two approaches:udNew sampling schemes have recently emerged that are able to perfectly reconstructudcertain classes of parametric signals that are not bandlimited but characterized by a finiteudnumber of parameters. In this context, we derive novel sampling schemes for piecewiseudsinusoidal and polynomial signals. In particular, we show that a piecewise sinusoidal signaludwith arbitrarily high frequencies can be exactly recovered given certain conditions. Theseudresults are applied to parametric multiview data that are not bandlimited.udWe also focus on the problem of extracting regions (or layers) in multiview imagesudthat can be individually rendered free of aliasing. The problem is posed in a multidimensionaludvariational framework using region competition. In extension to previousudmethods, layers are considered as multi-dimensional hypervolumes. Therefore the segmentationudis done jointly over all the images and coherence is imposed throughout theuddata. However, instead of propagating active hypersurfaces, we derive a semi-parametricudmethodology that takes into account the constraints imposed by the camera setup and theudocclusion ordering. The resulting framework is a global multi-dimensional region competition that is consistent in all the images and efficiently handles occlusions. We show theudvalidity of the approach with captured light fields. Other special effects such as augmentedudreality and disocclusion of hidden objects are also demonstrated.