Object Referring in Videos with Language and Human Gaze

摘要

We investigate the problem of object referring (OR) i.e. to localize a targetobject in a visual scene coming with a language description. Humans perceivethe world more as continued video snippets than as static images, and describeobjects not only by their appearance, but also by their temporal-spatialcontexts and motion features. Humans also gaze at the object when they issue areferring expression. Existing works for OR mostly focus on static images only,which fall short in providing many such cues. This paper addresses OR in videoswith language and human gaze. To that end, we present a new video dataset forOR, with 30, 000 objects over 5, 000 stereo video sequences annotated for theirdescriptions and gaze. We further propose a novel network model for OR invideos, by integrating appearance, motion, gaze, and spatial-temporalcontextual information all into one network. Experimental results shows thatour method effectively utilizes motion cues, human gaze, and spatial-temporalcontext information. Our method outperforms previous OR methods. The datasetand code will be made available.