CROSSLINK: Joint Understanding of Image and 3D Model Collections through Shape and Camera Pose Variations

摘要

Collections of images and 3D models hide in them many interestingrnaspects of our surroundings. Significant efforts have been devotedrnto organize and explore such data repositories. Most such efforts,rnhowever, process the two data modalities separately, and do not takernfull advantage of the complementary information that exist in differentrndomains, which can help to solve difficult problems in one byrnexploiting the structure in the other. Beyond the obvious differencernin data representations, a key difficulty in such joint analysis lies inrnthe significant variability in the structure and inherent properties ofrnthe 2D and 3D data collections, which hinders cross-domain analysisrnand exploration. We introduce CROSSLINK, a system for jointrnimage-3D model processing that uses the complementary strengthsrnof each data modality to facilitate analysis and exploration. We firstrnshow how our system significantly improves the quality of textbasedrn3D model search by using side information coming from anrnimage database. We then demonstrate how to consistently align thernfiltered 3D model collections, and then use them to re-sort imagerncollections based on pose and shape attributes. We evaluate ourrnframework both quantitatively and qualitatively on 20 object categoriesrnof 2D image and 3D model collections, and quantitativelyrndemonstrate how a wide variety of tasks in each data modality canrnstrongly benefit from the complementary information present in thernother, paving the way to a richer 2D and 3D processing toolbox.