How different mode of representation (video, audio, video+audio and written words) can influence the understanding and interpretation of AES

摘要

For every human being, it is essential to transmit information and advise other peopleabout their own affective epistemic states. On the other hand, it is also indispensable forevery person to be able to interpret the states of other people with whom they interact.By the term affective epistemic states (AES) we want to indicate all those states thatinvolve cognition, perception and feeling, or as Schroder suggests “states which involveboth knowledge and feeling” (Schroder, 2011).The aim of this paper is to show how different mode of representation (video, audio, acombination of audio+video and written words) can influence the understanding andinterpretation of AES. We also want to examine the effect of multimodality (using visualand auditory sensory modality simultaneously) compared to unimodality (using justvisual or just auditory sensory modality). Even if some studies investigated the area ofemotions and affective states, it is still very hard to find research that involves epistemicfeatures. More studies are essential in order to understand the mechanism about howAES are interpreted by humans.We conducted an experiment at the University of Gothenburg with 12 Swedishparticipants. Four recordings of first encounters face-to-face interaction, were displayedto each respondent. Each recording was shown in a different mode. The modes used forthe experiment consisted of a transcriptions (T), a video with audio (V+A), a videowithout audio (V) and an audio recording (A). The recordings were all about two peoplethat were meeting for the first time. We asked the respondents to identify which kinds ofAES were displayed by people participating in the recording. Respondents were asked tomotivate their answers.Several interesting outcomes have been observed. Participants were able to interpretdifferent AES when exposed to the same behavior in different modes. This means thatwhen the same behavior is displayed in different modes, the respondent’s perception isoften influenced in different ways. The same AES can be shown through vocal andgestural behaviors, and it can be perceived by visual, auditory or both modalitiestogether, depending on the modes displayed. We observed that AES are highlymultimodal and the majority of the times, different behaviors are perceived differently,depending on if they were shown by multimodal or unimodal modes.