Fusing Multimodal Human-Robot Communication using Deep Learning

摘要

To enable humanoid robots to share common house space with humans in a social environment, we need to develop effective and robust human-robot communication technology. In the present paper, we have developed gesture and speech based real time communication methodology with humanoid robot using deep learning based techniques. More specifically, for speech based communication, two architectures have been developed using Long Short Term Memory(LSTM) based Recurrent Neural Networks and Convolutional Neural Networks (CNN) for both Hindi and English speech. For gesture recognition and subsequent communication, a specially configured 3-D CNN architecture has also been developed. Further, a fusion framework has been used for comparing confidence scores of all the input modes so that the robot can take a decision for communication based on maximum confidence score. Rigorous experiments have been conducted with standard data sets as well as indigeneously created data sets. Performance results of different architectures have been presented and useful conclusions have been drawn about the effectiveness of using such multiple modes for the robot for decision reinforcement purposes. For conducting all the experiments the Humanoid Robot NAO has been used for evaluating the above methodology.