Eye4Ref: A Multimodal Eye Movement Dataset of Referentially Complex Situations

摘要

Eye4Ref is a rich multimodal dataset of eye-movement recordings collected from referentially complex situated settings where the linguistic utterances and their visual referential world were available to the listener. It consists of not only fixation parameters but also saccadic movement parameters that are time-locked to accompanying German utterances (with English translations). Additionally, it also contains symbolic knowledge - contextual - representations of the images to map the referring expressions onto the objects in corresponding images. Overall, the data was collected from 59 participants in three different experimental setups (86 systematically controlled sentence-image pairs and 2024 eye-movement recordings). Referential complexity was controlled by visual manipulations (e.g. number of objects in the scene, visibility of the target items, etc.), and by linguistic manipulations (e.g. the position of the disambiguating word in a sentence, of the relative clause attachment). This multimodal dataset - in which the three different sources of information, namely eye-tracking, language, and visual environment, are aligned - provides testing possibilities for various research questions not only from a linguistic but also from a computer vision perspective.