direction of look at understanding spoken language in multimodal conversational interactions

摘要

The present invention relates to improving the accuracy of understanding and / or resolving references to visual elements in a visual context associated with a computerized conversation system. The techniques described here leverage gesture input and / or voice input to improve understanding of spoken language in computerized conversation systems. Leveraging eye input and speech input enhances understanding of spoken language in conversational systems by improving the accuracy with which the system can resolve references or interpret a user's intention for visual elements in a visual context. In at least one example, the techniques describe eye tracking to generate eye input, speech input recognition, and extraction of eye features and lexical features from user input. based, at least in part, on look characteristics and lexical characteristics, user utterances directed at visual elements in a visual context can be resolved.