A study on method of gesture generation for multimodal dialogue system with a pen

摘要

We are now developing a problem solving and knowledge acquisition system based on Co reference between drill texts and dialogue with a teacher, focusing on first grade mathematics. The system offers multiple input and output channels such as natural language, gestures, and graphics. This paper discusses a method of generating gestures in cooperation with utterance generation. A gesture in this case means pointing of objects on the drill text or drawing pictures. It is represented by the movement of a pen, and is displayed as 3-D graphical data. We present Gesture Frame for a meaning description of gesture. First, the architecture of our system is summarized. Then the pointer sequence and gesture frame are defined in order to describe the structure and meaning of gesture. Lastly, results of the simulation are shown.