Information Enquiry Kiosk with Multimodal User Interface

摘要

A multimodal interactive dialogue automaton (kiosk) for self-service is presented in the paper.Multimodal user interface allow people to interact with the kiosk by natural speech, gestures additionally tothe standard input and output devices. Architecture of the kiosk contains key modules of speech processingand computer vision. An array of four microphones is applied for far-field capturing and recording of user'sspeech commands, it allows the kiosk to detect voice activity, to localize sources of desired speech signals, andto eliminate environmental acoustical noises. A noise robust speaker-independent recognition system isapplied to automatic interpretation and understanding of continuous Russian speech. The distant speech rec-ognizer uses grammar of voice queries as well as garbage and silence models to improve recognition accuracy.Pair of portable video-cameras are applied for vision-based detection and tracking of user's head and bodyposition inside of the working area. Russian-speaking talking head serves both for bimodal audio-visualspeech synthesis and for improvement of communication intelligibility by turning the head to an approachingclient. Dialogue manager controls the flow of dialogue and synchronizes sub-modules for input modalitiesfusion and output modalities fission. The experiments made with the multimodal kiosk were directed to cog-nitive and usability studies of human-computer interaction by different communication means.