Emotion recognition apparatus and method based on multimodal fusion

摘要

The present invention is implemented as a pre-learned artificial neural network and sequentially receives each of a depth sequence image including T consecutive frames, a column sequence image, and a color sequence image according to a frame order, and a frame applied according to a learned method A spatial encoder that sequentially extracts spatial features of each to obtain T depth feature maps, column feature maps, and color feature maps, respectively is sequentially approved, adds temporal features between feature maps sequentially applied according to the learned method, and acquires fused color spatiotemporal features by fusion decoding. Extracting 3D feature volumes from color sequence images according to the sequence, accumulating T fused color spatiotemporal features that are sequentially acquired to obtain spatiotemporal attention volumes, and combining 3D feature volumes and spatiotemporal attention volumes to obtain attention-enhancing feature volumes Including an attention volume acquisition unit and an emotion estimator that estimates and acquires emotional values from spatiotemporal attention volumes according to the learning method implemented with a pre-learned artificial neural network, multimodal images are fused to recognize emotions and time changes are reflected together It is possible to provide an emotion recognition apparatus and method capable of recognizing emotion very accurately.