A Multimodal Fusion Framework for Children's Storytelling Systems

摘要

Storytelling by a child, as a training activity, influences significantly a child's linguistic abilities, logic and thought process, imagination, and creativity. There presently exist many software-based storytelling applications. However, most are deemed incompatible or not suitable to Chinese children. Due to the limited vocabulary of pre-school and lower-level grade school children, speech-based and pen-based input models are considered the most effective way of input. But now there is not an effective multimodal mode to solve the problem for children's storytelling systems. In this paper, we propose a multi-modal fusion framework that utilizes pen and speech techniques to incorporate both context information and linguistic attributes of the Chinese language into the design. Based on the proposed framework, we formulated specific methods of integration, and developed a prototype for our proposed system.