Enhancing Natural Language Understanding through Cross-Modal Interaction: Meaning Recovery from Acoustically Noisy Speech

摘要

Cross-modality between vision and language is a key component for effective and efficient communication, and human language processing mechanism successfully integrates information from various modalities to extract the intended meaning. However, incomplete linguistic input, i.e. due to a noisy environment, is one of the challenges for a successful communication. In that case, incompleteness in one channel can be compensated by information from another one (if available). In this paper, by employing a visual-world paradigm experiment, we investigated the dynamics between syntactically possible gap fillers for incomplete German sentences and the visual arrangements and their effect on overall sentence interpretation.