Multimodal interaction of auditory spatial cues and passive whole-body movement through virtual space

摘要

When multisensory stimulation is coordinated within a comprehensive simulation, the resulting multimodal display can become so entirely convincing that it can create an experience of the observer's travel through a virtual environment. A great deal of attention has been paid to coordinated display within the auditory and visual modalities, but even the best of such bimodal simulations may fail to produce satisfying results when the user is intended to move through a virtual world. Although visual information strongly affects self motion perception, there are situations in which auditory cues alone are available to induce perceived self motion in observers. This paper will summarize our recent research efforts concerning the interaction between auditory spatial information and passive whole-body movement in self motion perception. Firstly, we describe the effect of postural information on the perceived velocity of moving sound sources, regarding which our obtained results suggested that the strongest multimodal interaction occurs when auditory information and postural variation are well matched. Secondly, the temporal synchrony between passive whole-body motion and auditory spatial information was investigated via a multimodal time-order judgment task. Our results suggested that sensory integration of auditory motion cues with whole-body movement cues could occur over an increasing range of intermodal delays as the velocity of virtual sound sources motion was decreased. The results of these two studies may be of general interest to those researching multimodal interaction, but may also be useful to those developing next generation multimodal display systems.