Gaze for understanding spoken language in conversational dialogue in multiple modes

摘要

Improved accuracy in understanding and / or resolving instructions to visual elements within the visual context associated with a computerized conversational system is described. The technique described in this article uses gaze input along with gestures and / or input to improve the understanding of spoken language in conversational systems. This is due to improving the accuracy with which the system can resolve instructions regarding visual elements within the visual context or interpret user intent. In at least one example, the techniques of this paper describe tracking gaze to generate gaze input, recognizing speech input, and extracting gaze and lexical features from user input. A user utterance directed to a visual element within the visual context can be resolved based at least in part on the gaze and lexical features.