Generating and streaming immersive sports video content

摘要

Stereoscopic 3D videos have already become popular in movie theaters with most productions being released in this format. More recently, with the availability of commodity Virtual Reality (VR) products, immersive video content is receiving even more interest. A wide spread adoption of immersive devices and displays is hindered by the lack of content that matches the user expectations. Producing immersive videos is far more costly and time-consuming than regular 2D videos, which makes it challenging and thus rarely attempted, especially for live events, such as sports games. In addition, immersive content needs to be adapted for viewing on different displays/devices. To address these challenges, we first propose a new system for 3D video streaming that provides automatic depth adjustments as one of its key features. Our system takes into account both the content and the display type in order to customize 3D videos and optimize the viewing experience. Our stereoscopic video streaming system was implemented, deployed and tested with real users. Results show that between 60% to 70% of the shots can benefit from our system and more than 25% depth enhancement can be achieved. Next, we propose a novel, data-driven method that converts 2D videos to 3D by transferring depth information from a database of similar 3D videos. Our method then reconstructs the depth map while ensuring temporal coherency using a spatio-temporal formulation of Poisson reconstruction. Results show that our method produces high-quality 3D videos that are almost indistinguishable from videos shot by stereo cameras, while achieving up to 20% improvement in the perceived depth compared to the current state-of-the-art method. Furthermore, we extend our work in the direction of VR, and propose using video feeds from regular broadcasting cameras to generate sports VR content. We generate a wide-angle panorama by utilizing the motion of the main camera. We then use various techniques to remove the parallax, align all video feeds, and overlay them on the panorama. Subjective studies show that our generated content provides an immersive experience similar to ground-truth content captured using a 360 camera, with most subjects rating their sense of presence from Good to Excellent.