A Corpus for Investigating the Multimodal Nature of Multi-Speaker Spontaneous Conversations - EVA Corpus

摘要

Multimodality and multimodal communication is a rapidly evolving research field addressed by scientists working in various perspectives, from psycho-sociological fields, anthropology and linguistics, to communication and multimodal interfaces, companions, smart homes and ambient assisted living etc. Multimodality in human-machine interaction is not just an add-on or a style of information representation. It goes well beyond semantics and semiotic artefacts. It can significantly contribute to representation of the information as well as in interpersonal and textual function of communication. The study in this paper is a part of an ongoing effort in order to empirically investigate in detail relations between verbal and co-verbal behavior expressed during multi-speaker highly spontaneous and live conversations. It utilizes a highly multimodal approach for investigating into relations between the traditional linguistic (such as: paragraphs, sentences, sentence types, words, POS tags etc.) and prosodic features (such as: phrase breaks, prominence, durations, and pitch), and paralinguistic features traditionally interpreted as non-verbal communication or co-verbal behavior (such as: dialog role, semiotic classification of behavior, emotions, facial expressions, head movement, gaze, and hand gestures). The main motivation for this study is to be able to understand especially the informal nature of human-human communication, and to create co-verbal resources for automatic synthesis of highly natural co-verbal behavior from un-annotated text and expressed through embodied conversational agents. The EVA corpus designed by a novel EVA annotation scheme represents a rich empirical resource for performing such studies in conversational phenomena that manifest themselves in highly spontaneous face-to-face conversations. A preliminary analysis regarding emotions within conversations has been also conducted and presented in the paper.