Automatic social signal analysis: Facial expression recognition using difference convolution neural network

摘要

Facial expression is one of the most powerful social signals for human beings to convey emotion and intention, hence automatic facial expression recognition (FER) has wide applications in human computer interaction and affective computing, it has attracted an increasing attention recently. Researches in this field have made great progress especially with the development of deep learning method. However, FER remains a challenging task due to individual differences. To address the issue, we propose a two-stage framework based on Difference Convolution Neural Network (DCNN) inspired by the facial expression's nonstationary nature. In the first stage, the neutral expression frame and fully expression frame are automatically picked out from the facial expression sequences using a binary Convolution Neural Network (CNN). Then in the second stage, an end-to-end DCNN is proposed to classify the six basic facial expressions using the difference information between the neutral expression frame and the fully expression frame. Experiments have been conducted on the CK+ and BU-4DFE datasets, and the results show that the proposed framework delivers a promising performance (95.4% on the CK+ dataset and 77.4% on the BU-4DFE). Moreover, the proposed method is also successfully applied to analyze the student's affective state in an E-learning environment which suggests that it has strong potential to analyze nonstationary social signals. (C) 2019 Elsevier Inc. All rights reserved.