Immersive video applications grow faster for users to freely navigate within a virtualized 3D environment forentertainment, productivity, training, etc. Fundamentally, such system can be facilitated by an interactiveGigapixel Video Streaming (iGVS) platform from array camera capturing to end user interaction. This interactivesystem demands a large amount of network bandwidth to sustain the reliable service provisioning, hinderingits massive market adoption. Thus, we propose to segment the gigapixel scene into non-overlapped spatialtiles. Each tile only covers a sub-region of the entire scene. One or more tiles will be used to represent aninstantaneous viewport interested by a speci c user. Tiles are then encoded at a variety of quality scales usingvarious combinations of spatial, temporal and amplitude resolutions (STAR), which are typically encapsulatedinto temporally-aligned tile video chunks (or simply chunks). Chunks at di erent quality level can be processedin parallel for real-time purpose. With such setup, diverse chunk combinations can be simultaneously accessed byheterogeneous user per its request, and viewport-adaptation based content navigation in an immersive space canbe also realized by adapting multiscale chunks properly, under the bandwidth constraints. A serial computationalvision models measuring the perceptual quality of viewport video in terms of its quality scales, adaptation factors,as well as the peripheral vision thresholds, are devised to prepare and guide the chunk adaptation for the bestperceptual quality index. Furthermore, in response to the time-varying network, a deep reinforcement learning(DRL) based adaptive real-time streaming (ARS) scheme is developed, by learning the future decision fromthe historical network states, to maximize the overall quality of experience (QoE) in a practical Internet-basedstreaming scenario. Our experiments have revealed that averaged QoE can be improved by about 60%, andits standard deviation can be also reduced by 30%, in comparison to the popular Google congestion controlalgorithm widely adopted in existing system for adaptive streaming, demonstrating the e ciency of our multiscaleaccelerated iGVS for immersive video application.
展开▼
