Designing synthesis in image and word: Multimodal composing in a Japanese university context.

摘要

In a semiotic climate increasingly characterized by hybrid media and multiple modes of communication, how is it that coherence may be created in electronic texts? In particular, how might a multimedia author approach the designing of textual meaning in such ways that potential meanings are not merely multiplied to a polysemous effect, but rather synthesized, so that efficacious relationships and alignments of meaning are created among the diverse meaning-making resources at hand? This study addressed these questions in the creative context of the practice digital storytelling, in which nine Japanese undergraduate students were engaged. The research followed and documented these participants' language-image design processes over the course of four months, employing a hybrid case study methodology and applying conceptual and analytic tools from the fields of semiotics, linguistics, and cognitive science. Findings of the investigation suggest that the key to managing 'ecologies of meaning' in multimodal composition may lie in the capacity to create multi-level designs-within-designs, which needs an integrative body of adaptive knowledges of the several types: (1) awareness of what kinds of meanings different modes tend to afford and how the affordances of different modes may interact; (2) sensitivity to conceptual commonalities that cross modal boundaries, like metaphorical relationships; (3) recognition of opportunities to create more apt resources for meaning making by re-crafting existing ones; and (5) understanding of how a perceived audience or audiences might constrain design choices. Pivotal, too, is the role of context, according to which these knowledges may vary significantly.