Enhancing Viewing Experience of Generated Visual Storylines for Promotional Videos

摘要

Visual storyline generation is the problem of selecting and sequencing a set of visual materials (i.e. images and video clips) to produce a video to elicit certain cognitive or emotional responses from viewers. In this paper, we enhance the viewing experience of generated visual storylines with the Shot Composition, Selection and Plotting (ShotCSP) approach. Designed for generating promotional videos in ecommerce settings, ShotCSP considers three key film-making principles into the visual storyline generation pipeline: a) proximity-aware scene transition, b) sound logic flow, and c) graphic discontinuity. We propose two novel metrics to enhance viewing experience: 1) Semantic Distance, which measures how related a shot is to the product being promoted; and 2) Salient Region Ratio, which estimates attention to product details in a shot. Through large-scale user evaluation involving 1,748 pairwise comparisons against five state-of-the-art approaches, ShotCSP achieves significantly improved viewing experience. It is a promising approach to enable AI generated promotional videos to benefit e-commerce businesses.