Comprehensive approach for solving multimodal data analysis problems based on integration of evolutionary, neural and deep neural network algorithms

摘要

In this work we propose the comprehensive approach for solving multimodal data analysis problems. This approach involves multimodal data fusion techniques, multi-objective approach to feature selection and neural network ensemble optimization, as well as convolutional neural networks trained with hybrid learning algorithm that includes consecutive use of the genetic optimization algorithm and the back-propagation algorithm. This approach is aimed at using different available channels of information and fusing them at data-level and decision-level. The proposed approach was tested on the emotion recognition problem. SAVEE database was used as the raw input data, containing visual markers dataset, audio features dataset, and the combined audio-visual dataset. The best emotion recognition accuracy achieved with the proposed approach on visual markers data is 65.8%, on audio features data -52.3%, on audio-visual data - 71%.