Liar, liar, hands on fire: What gesture -speech asynchrony reveals about thinking.

摘要

An examination of the relationship between speech and gesture exploits the fact that gestures are unwitting and sensitive manifestations of speaker-internal thought processes. By looking at the synchrony between modalities, gesture may reveal more about thinking than may be expressed by speech. Previous models of language production that include gesture have focused on the expression of a singular representation or idea. This project analyzes gesture and speech when multiple construals of an event are simultaneously expressed. Asynchrony between channels is investigated considering both the meanings expressed in each modality as well as the temporal alignment of speech and gesture. To induce multiple representations of a scene, I ask participants to view a cartoon involving a cat and a bird. Prior to retelling the cartoon to a naive listener, I instruct the participants to deceive their interlocutors by misreporting portions of the cartoon content. I then consider what modality conveys viewed versus instructed information, how often the channels synchronize, and the implications of semantic asynchrony on temporal synchrony. In the second part of the dissertation, I probe gesture-speech asynchronies in data where multiple construals of an event are naturally occurring and removed from deception. Via these currents experiments, I argue that gesture and speech remain a unified system when multiple representations of an event are active in a speaker's mind. While no current model of language production includes gesture-speech mismatches, at their heart, these expressions may not be so different from typical gesture-speech pairings.