When scenes and words collide: Irrelevant background scenes modulate neural responses during lexical decisions.

摘要

Usually, linguistic operations and visual-perceptual operations are studied separately in domain-specific experimental paradigms. However, there has been evidence that language and image processing interact behaviorally in a purely linguistic task (V?μ & Wolfe, VSS 2014). Here we test whether brain responses during lexical decisions are modulated by an irrelevant, visual background scene, via analysis of EEG signals. Participants were presented with a background scene and a location cue before a string of letters appeared at the pre-cued scene location. The sole task was to decide whether the letter string formed a word or non-word. Words could either be congruent with the scene (a??SOAPa?? on sink), semantically incongruent (a??EGGa?? on sink), syntactically incongruent (a??SOAPa?? on towel rack a?? i.e., semantically congruent but it in a wrong relative location), or double-incongruent (a??EGGa?? on towel rack). We found that words that were semantically incongruent with respect to the background scene triggered a negative deflection compared to the consistent words about 400ms after word onset. In the language domain, this N400 response is known to signal difficulties in the semantic integration of a word with its sentence context. Interestingly, this incongruity effect between scene and words was more pronounced on left- rather than right-hemispheric electrodes, possibly due to the involvement of linguistic processes. Semantically congruent words presented in improbable scene locations, on the other hand, did not significantly affect brain responses. Thus, the syntactic placement of a word on an irrelevant background scene did not modulate neural responses to the extent that semantically incongruent words did. We conclude that a brief visual scene preview a?? even if task-irrelevant a?? automatically interacts with linguistic operations on at least the semantic processing level. Therefore, language and visual scene processing may share common parsing mechanisms that are efficiently integrated to function as a unitary whole.