A song of scenes & sentences: signatures of shared cortical resources between visual perception and language revealed by representational similarity analysis

摘要

Previous imaging studies, investigating the domain specificity of cortical networks, have indicated some common principles of processing across different cognitive functions and therefore shared cortical resources, e.g. the processing of hierarchical structures ("syntax") or contextual meaning ("semantics"). Whereas the majority of research focused on language and music, recent studies also emphasized comparable principles in visual perception. However, little is known about the degree of possibly shared cortical resources between vision and language. To overcome this existing gap, we created a paradigm consisting of two modalities, visual (natural scenes) and auditory (sentences) stimuli, equally divided into consistent, semantically inconsistent, and syntactically inconsistent. Twenty participants either viewed images or listened to sentences while BOLD-responses were recorded. We assessed cortical activation patterns for semantic and syntactic language processing, applying the general linear model in each participant's native space, thus creating participant and semantic/syntax specific functional ROIs (pfROIs). Subsequently we conducted a representational similarity analysis (RSA) within those pfROIs including activation patterns from all conditions and modalities to investigate the relationship between activation patterns of language and visual perception more closely. Both language conditions activated the expected left-lateralized networks, compromising IFG, STS/MTG and IPL (semantic) and IFG, as well as STS/STG (syntax). RSA in all pfROIs revealed distinguishable patterns between modalities. Focusing on the patterns more closely we found highest similarities across modalities for both, semantic and syntactic processing in their respective pfROIs. In particular, the semantic pfROIs showed highest similarity between the activation patterns for semantic processing of language and vision, whereas syntactic processing revealed most similar activation patterns in the syntactic pfROIs. These results underline a specific and distinct processing for semantic and syntax, additionally giving a first insight on common principles between vision and language, as the resulting activation patterns for either semantic or syntax processing were most similar across modalities.