Survey and challenges of story generation models - A multimodal perspective with five steps: Data embedding, topic modeling, storyline generation, draft story generation, and story evaluation

摘要

The story is the description of events in chronological order that have occurred between people. By delivering facts to the people reading the story, it enables them to feel emotions. Such a story is composed using the following method: each event is analyzed and a storyline is composed, which becomes a skeleton text by linking relationships between major events. As the content of users becomes more diverse, multimodal story composition has become more essential than unimodal text-based story composition. This paper discusses modality integration based on multimodal data types and type conversion for multimodal story composition. We propose a story-graph model to create a story based on the integrated analysis of various modal data. In terms of architecture, the proposed multimodal storytelling model consists of modal data and a topic modeling module that performs clustering based on cross-modal similarities and extracts a topic of clustered modalities. From the perspective of utilization, to visualize a story-graph, the proposed model summarizes nodes with a representative image. Furthermore, the latest techniques are discussed with respect to five main modules and twelve sub modules for story composition. Lastly, problems that can become issues when composing multimodal story modules are explained.