Why Are Multimodal Systems so Difficult to Build? - About the Difference between Deictic Gestures and Direct Manipulation

摘要

By integrating conversationally used natural language with graphical interfaces, gestural interaction looses the simplicity of direct manipulation. Input in one channel may change the meaning of input in another channel rather drastically. This introduces the wait problem, i.e. the problem of when and how long the system has to wait for input in other channels, before it triggers an action. Additionally, pointing devices and feedback may cause obtrusive effects on the natural synchronisation of deictic gestures and deictic expressions. Certain uses of gestures are observed that are distinct from natural communication and direct manipulation as well, e.g. focusing gestures. The discussion of these problems is based on our experience with the multimodal prototypes MOFA and TALKY that we implemented in different variations, each showing a slightly different style of interaction. The notion of passive and active gesture forms is introduced as well as the notion of active and passive objects. Incremental natural language interpretation and the provision of incremental and preliminary feedback turn out to be important challenges for the upcoming technology of multimodal interfaces.