A method of pen motion generation for multimodal dialogue system

摘要

We are now developing a problem solving system based on co-reference between drill texts and dialogue with a teacher, focusing on first grade mathematics. The system offers multiple input and output channels such as natural language, gestures, and graphics. This paper discusses a method of generating pen motions in cooperation with utterance generation. A gesture in this case means pointing of objects on the drill text or drawing pictures. It is represented by the movement of a pen, and is displayed as 3-D graphical data. We present Pen Motion Frame for a meaning description of gesture. First, the architecture of our system is summarized. Then the pointer sequence and gesture frame are defined in order to describe the structure and meaning of gesture. Lastly, results of the simulation are shown.