Landmark guidance independent spatio-channel attention and complementary context information based facial expression recognition

摘要

Attention based convolutional neural networks(CNNs) for facial expression recognition (FER) apply atten-tion that is uniform across either spatial dimensions or channel dimensions or both spatial and channel dimensions. However, there are many issues viz. (i) in the presence of occlusions and pose variations, different channels respond differently, (ii) the response intensity of a channel differ across spatial lo-cations, (iii) attention is defined based on external sources like landmark detectors and (iv) features used from pretrained face recognition (FR) model to complement the attention branch contain redun-dant information. To overcome these issues, an end-to-end architecture for FER is proposed in this work. This architecture obtains both local and global attention per channel per spatial location through a novel spatio-channel attention net (SCAN), without seeking any information from the landmark detectors. SCAN is complemented by a complementary context information (CCI) branch that builds expression represen-tation from the pretrained FR features. Redundancies in FR features are eliminated by using efficient channel attention (ECA). The representation learnt by the proposed architecture is robust to occlusions and pose variations. This is demonstrated by the state-of-the-art performance of the proposed model on in-the-wild datasets including AffectNet, FERPlus, RAF-DB, SFEW and FED-RO. Further, the proposed ar-chitecture also reports superior performance on in-lab datasets (CK+, Oulu-CASIA and JAFFE) and a couple of constructed face masked datasets resembling masked faces in COVID-19 scenario. Codes are publicly available at https://github.com/1980x/SCAN-CCI- FER .(c) 2021 Elsevier B.V. All rights reserved.