How do non-expert users exploit simultaneous inputs in multimodal interaction?

摘要

This paper evaluates scenario based user tests of speech-centric multimodal interaction on a small mobile terminal. Non-expert users solved tasks in the tourist guide domain using a functional multimodal PDA-based application. The tasks required pen and speech input, but the users were free to choose either sequential or simultaneous pen and speech input at each step in the dialogue. Multimodal interfaces are still a novelty to most users so we had to explain this functionality to the test users. The format of the introduction had a noticeable effect on user behaviour. Users who had seen a video demonstration used simultaneous pen and speech input more often than the users who had had a text only introduction even if the same information was present in both formats. 9 of 14 subjects who had seen the video demo, applied simultaneous pen and speech input instantly. We therefore claim that people will use simultaneous multimodal input when they have been properly introduced to this functionality. However, simultaneous use of pen and speech may impose an extra cognitive load, at least until people get familiar with this kind of interface. The users considered the multimodal interaction attractive and expressed that they enjoyed the freedom of choosing input mode at each step in the dialogue.