Egocentric visitor localization and artwork detection in cultural sites using synthetic data

摘要

Computer vision and machine learning can be used in cultural heritage to augment the experience of visitors during the exploration of the cultural site, as well as to assist its management. To achieve such goals, two fundamental tasks should be addressed, i.e., localizing visitors and recognizing the observed artworks. Wearable cameras offer a convenient setting to address both tasks through the analysis of images acquired from the visitors' points of view. However, the engineering of approaches to address such tasks generally requires large amounts of labeled data. We propose a tool which can be used to collect and automatically label synthetic visual data suitable to study image-based localization and artwork detection. The tool simulates a virtual agent navigating the 3D model of a real cultural site and automatically captures video frames along with the related ground truth camera poses and semantic masks indicating the position of artworks. We generate a dataset of synthetic images starting from the 3D model of a museum located in Siracusa, Italy. The experiments suggest that the proposed tool allows to drastically reduce the effort needed to collect and label data, providing a means to generate large-scale datasets suitable to study localization and artwork detection in cultural sites. (C) 2020 Elsevier B.V. All rights reserved.