Robot recognizing humans intention and interacting with humans based on a multi-task model combining ST-GCN-LSTM model and YOLO model

摘要

It is hoped that the robot could interact with the human when the robots help us in our daily lives. And understanding humans' specific intention is the first crucial task for human-robot interaction. In this paper, we firstly develop a multi-task model for recognizing humans' intention, which is composed of two sub-tasks: human action recognition and hand-held object identification. For the front subtask, an effective ST-GCN-LSTM model is proposed by fusing the Spatial Temporal Graph Convolutional Networks and Long Short Term Memory Networks. And for the second subtask, the YOLO v3 model is adopted for the hand-held object identification. Then, we build a framework for robot interacting with the human. Finally, these proposed models and the interacting framework are verified on several datasets and the testing results show the effectiveness of the proposed models and the framework. (c) 2020 Elsevier B.V. All rights reserved.