Added Value of Gaze-Exploiting Semantic Representation to Allow Robots Inferring Human Behaviors

摘要

This article presented the extension and improvement of our semantic-based learning method by including the information of different input sources such as one external camera and a gaze camera. This is possible due to our (realistic) proposed setup, which enables the possibility to assess the valuable input of the gaze information to enhance the robot capabilities in real-world scenarios. Our proposed system has the capability to adapt according to the available input information of the robot by switching between the observed object properties ObjActedOn and ObjectSeen. The obtained semantic rules have the important characteristic that can be used in different scenarios and across different datasets to correctly infer human activities with an average recognition of 84% when combining the external and gaze information validated in a humanoid robot. We have validated the robustness of our system using videos from the CMU-MMAC dataset, which has a similar setting as the one proposed. In this case, we enhance the segmentation and recognition of activities from 54% up to 86% for the new scenario of scrambling eggs, exploiting the benefits of our general semantic models using the combination of the egocentric and external cameras to infer human activities.