Unstructured Video-Based Rendering: Interactive Exploration of Casually Captured Videos

摘要

We present an algorithm designed for navigating around a performancernthat was filmed as a “casual” multi-view video collection:rnreal-world footage captured on hand held cameras by a few audiencernmembers. The objective is to easily navigate in 3D, generatingrna video-based rendering (VBR) of a performance filmedrnwith widely separated cameras. Casually filmed events are especiallyrnchallenging because they yield footage with complicatedrnbackgrounds and camera motion. Such challenging conditions precludernthe use of most algorithms that depend on correlation-basedrnstereo or 3D shape-from-silhouettes.rnOur algorithm builds on the concepts developed for the explorationrnof photo-collections of empty scenes. Interactive performerspecificrnview-interpolation is now possible through innovations inrninteractive rendering and offline-matting relating to i) modeling thernforeground subject as video-sprites on billboards, ii) modeling thernbackground geometry with adaptive view-dependent textures, andrniii) view interpolation that follows a performer. The billboards arernembedded in a simple but realistic reconstruction of the environment.rnThe reconstructed environment provides very effective visualrncues for spatial navigation as the user transitions between viewpoints.rnThe prototype is tested on footage from several challengingrnevents, and demonstrates the editorial utility of the whole systemrnand the particular value of our new inter-billboard optimization.