The Effect of a Concurrent Working Memory Task andTemporal Offsets on the Integration of Auditory andVisual Speech Information

摘要

Audiovisual speech perception is an everyday occurrence of multisensory integration. Conflicting visualspeech information can influence the perception of acoustic speech (namely the McGurk effect), and audi-tory and visual speech are integrated over a rather wide range of temporal offsets. This research examinedwhether the addition of a concurrent cognitive load task would affect the audiovisual integration in a McGurkspeech task and whether the cognitive load task would cause more interference at increasing offsets. Theamount of integration was measured by the proportion of responses in incongruent trials that did not cor-respond to the audio (McGurk response). An eye-tracker was also used to examine whether the amount oftemporal offset and the presence of a concurrent cognitive load task would influence gaze behavior. Resultsfrom this experiment show a very modest but statistically significant decrease in the number of McGurkresponses when subjects also perform a cognitive load task, and that this effect is relatively constant acrossthe various temporal offsets. Participant's gaze behavior was also influenced by the addition of a cognitiveload task. Gaze was less centralized on the face, less time was spent looking at the mouth and more timewas spent looking at the eyes, when a concurrent cognitive load task was added to the speech task.