Relating Audio-Visual Events Caused by Multiple Movements: In the Case of Entire Object Movements and Sound Location Changes

摘要

Relating audio-visual events is important for constructing for an artificial intelligent system, which can acquire the audio-visual knowledge of movement through active observation without teaching. This paper proposes a method for relating multiple audiovisual events observed by a camera and a microphone according to general laws without object-specific knowledge (including the case of entire object movements and sound location changes). As corresponding cues, we use Gestalt's grouping law; simultaneity of the occurrence of the sound and the change in movement or the same motion starting, similarity of repetition between sound and movement. Based on the correlation coefficient between auditory and visual sequence, the component of frequency at sound onset is related to the short-term space-time invariants (STSTI) of movement. We experimented in the real environment and obtained satisfactory results showing the effectiveness of the proposed method.