(Pea)nuts and bolts of visual narrative: Structure and meaning in sequential image comprehension

摘要

Just as syntax differentiates coherent sentences from scrambledword strings, the comprehension of sequential images must alsouse a cognitive system to distinguish coherent narrative sequencesfrom random strings of images. We conducted experiments analo-gous to two classic studies of language processing to examine thecontributions of narrative structure and semantic relatedness toprocessing sequential images. We compared four types of comicstrips: (1) Normal sequences with both structure and meaning, (2)Semantic Only sequences (in which the panels were related to acommon semantic theme, but had no narrative structure), (3) Struc-tural Only sequences (narrative structure but no semantic related-ness), and (4) Scrambled sequences of randomly-ordered panels.In Experiment 1, participants monitored for target panels insequences presented panel-by-panel. Reaction times were slowestto panels in Scrambled sequences, intermediate in both StructuralOnly and Semantic Only sequences, and fastest in Normal sequences.This suggests that both semantic relatedness and narrative structureoffer advantages to processing. Experiment 2 measured ERPs to allpanels across the whole sequence. The N300/N400 was largest topanels in both the Scrambled and Structural Only sequences, inter-mediate in Semantic Only sequences and smallest in the Normalsequences. This implies that a combination of narrative structureand semantic relatedness can facilitate semantic processing ofupcoming panels (as reflected by the N300/N400). Also, panels in the Scrambled sequences evoked a larger left-lateralized anteriornegativity than panels in the Structural Only sequences. This local-ized effect was distinct from the N300/N400, and appeared despite'the fact that these two sequence types were matched on localsemantic relatedness between individual panels. These findingssuggest that sequential image comprehension uses a narrativestructure that may be independent of semantic relatedness. Alto-gether, we argue that the comprehension of visual narrative isguided by an interaction between structure and meaning.