Reaction time for two types of semantically related gesture and sentence pairs

摘要

The aim of this study was to investigate in an EEG experiment the reaction time of the participants deciding on the semantic congruency of sentences with subsequent pictures of symbolic or iconic hand gestures. More specifically, we investigated the reaction time employed by the participants to determine the semantic congruence of sentences and gesture pictures which are semantically related in different ways. In one case the gesture conveys the same content as the sentence, in the second case it provides additional content to the sentence. The hypothesis that we wanted to test was that the reaction time would be shorter in the latter case than in the former since preceding corpus-based studies have indicated that speech and gestures conveying the same information are more temporally synchronous than speech and gestures which convey complementary information. Therefore, we expect that it will be harder to process semantic congruency for gestures which usually are synchronous to speech if they follow the sentences they are related to than in the case where gestures provide additional information. Our study confirms the hypothesis showing that not only was the participants reaction time shorter in the case of gestures which provided additional information to sentences, but also that the accuracy in the semantic congruency task was higher.