3D Ken Burns Effect from a Single Image

摘要

The Ken Burns effect allows animating still images with a virtual camera scanand zoom. Adding parallax, which results in the 3D Ken Burns effect, enablessignificantly more compelling results. Creating such effects manually is timeconsumingand demands sophisticated editing skills. Existing automaticmethods, however, require multiple input images from varying viewpoints.In this paper, we introduce a framework that synthesizes the 3D Ken Burnseffect from a single image, supporting both a fully automatic mode and an interactive mode with the user controlling the camera. Our framework firstleverages a depth prediction pipeline, which estimates scene depth that issuitable for view synthesis tasks. To address the limitations of existing depthestimation methods such as geometric distortions, semantic distortions, andinaccurate depth boundaries, we develop a semantic-aware neural networkfor depth prediction, couple its estimate with a segmentation-based depthadjustment process, and employ a refinement neural network that facilitatesaccurate depth predictions at object boundaries. According to this depthestimate, our framework then maps the input image to a point cloud andsynthesizes the resulting video frames by rendering the point cloud from thecorresponding camera positions. To address disocclusions while maintaininggeometrically and temporally coherent synthesis results, we utilize contextawarecolor- and depth-inpainting to fill in the missing information in theextreme views of the camera path, thus extending the scene geometry of thepoint cloud. Experiments with a wide variety of image content show thatour method enables realistic synthesis results. Our study demonstrates thatour system allows users to achieve better results while requiring little effortcompared to existing solutions for the 3D Ken Burns effect creation.