A robot that can carry out a natural-language instruction has been a dreamsince before the Jetsons cartoon series imagined a life of leisure mediated bya fleet of attentive robot helpers. It is a dream that remains stubbornlydistant. However, recent advances in vision and language methods have madeincredible progress in closely related areas. This is significant because arobot interpreting a natural-language navigation instruction on the basis ofwhat it sees is carrying out a vision and language process that is similar toVisual Question Answering. Both tasks can be interpreted as visually groundedsequence-to-sequence translation problems, and many of the same methods areapplicable. To enable and encourage the application of vision and languagemethods to the problem of interpreting visually-grounded navigationinstructions, we present the Matterport3D Simulator -- a large-scalereinforcement learning environment based on real imagery. Using this simulator,which can in future support a range of embodied vision and language tasks, weprovide the first benchmark dataset for visually-grounded natural languagenavigation in real buildings -- the Room-to-Room (R2R) dataset.
展开▼
