The way of multisensory spatial processing with audio-visual speech stimuli differs in single and bilateral visual presentations

摘要

Ventriloquism is defined as a shift of perceptual location of a sound source toward a synchronized visual stimulus. Previous studies lead to the conclusion that the size of the ventriloquism effect is regulated by physical, not cognitive, factors. However, this conclusion is based upon simplified experimental designs which typically entail pairs of single audio and visual stimuli. Such designs do not capture our responses to a multisensory world where many signals impact various sensory modalities. We examined the hypothesis that cognitive factors, as well as physical ones, modulate the ventriloquism effect in complex situations. We used audio-visual speech stimuli in two experiments involving simplified (Experiment 1) and complex (Experiment 2) designs. Experiment 1 involved presentation of one movie of a face and one voice whereas Experiment 2 involved presentation of two bilateral movies and one voice. In both experiments a cognitive factor, namely congruency of speech and visual stimuli, was varied (congruent, incongruent). In both experiments visual stimuli appeared on a central CRT monitor, whereas auditory stimuli were presented from 13 positions, created by left (L)- right (R) phase differences. Participants judged if the location of an auditory source was left or right of a central fixation cross (on the monitor). In Experiment 1, we found no differences due to cognitive factors. Consistent with previous finding, the auditory localization bias did not differ as a function of congruency. In Experiment 2, we manipulated physical saliency between bilateral visual stimuli as a physical factor, as well as congruency of audio-visual syllables as a cognitive factor. In this experiment, both visual stimulus salience and audio-visual congruency elicited relatively large auditory localizations biases. In conclusion, these experiments show that a cognitive factor affects the way audio-visual spatial information is integrated in a more complex, real world, situation.