System and process for generating 3D video textures using video-based rendering techniques

摘要

A system and process for generating a 3D video animation of an object referred to as a 3D Video Texture is presented. The 3D Video Texture is constructed by first simultaneously videotaping an object from two or more different cameras positioned at different locations. Video from, one of the cameras is used to extract, analyze and synthesize a video sprite of the object of interest. In addition, the first, contemporaneous, frames captured by at least two of the cameras are used to estimate a 3D depth map of the scene. The background of the scene contained within the depth map is then masked out, and a clear shot of the scene background taken before filming of the object began, leaving just the object. To generate each new frame in the 3D video animation, the extracted region making up a frame of the video sprite is mapped onto the previously generated 3D surface. The-resulting image is rendered from a novel viewpoint, and then combined with a flat image of the background which has been warped to the correct location. In cases where it is anticipated that the subject could move frequently, the foregoing part of the procedure associated with estimating a 3D depth map of the scene and extracting the 3D surface representation of the object is performed for each subsequent set of contemporaneous frames captured by at least two of the cameras.