Methods of Multimodal Data Fusion and Forming Latent Representation in the Human Aggression Recognition Task

摘要

In recent years, the number of users of socio-cyber-physical systems, smart spaces, and the Internet of things systems has increased. This fact actualizes the problem of automatic recognition of destructive behavior of users, such as aggression. Users behavior in socio-cyber-physical systems can be represented in different modalities: facial expressions, motor activity of human body, non-verbal speech behavior, verbal speech behavior. This work considers possible methods and conceptual models of data fusion of various modalities in human aggression recognition task. All the considering methods are based on the transfer learning approach and multimodal data fusion, which forms a latent space. Three methods of multimodal data fusion are described: the first method is based on principal component analysis and singular value decomposition, the second method is based on a neural net-work (NN) that process data of different modalities simultaneously, and the third method is based on an NN that process data of different modalities separately. The conceptual models and mathematical description of the considering methods as well as key aspects of data processing and the main features of the forming la-tent spaces are provided.