Convolutional neural networks emotion recognition and blink characteristics analysis for operator state estimation

摘要

The paper deals with the problem of operator’s state estimating [1]. For this purpose various approaches based on using deep convolutional neural networks [2, 3] are proposed. The approach using automatic emotion recognition methods and analysis of the blink characteristics are considered in the most detail. Extensive research is being done into the emotional state and attention concentrating. On the trained neural network there was conducted an experiment to estimate the operator’s state. During the experiment 11 video records of the operator’s face received during operator performing the flight task on the flight simulator [4] were processed. In the normal operator’s state during the piloting there were 2 emotions observed – sadness and a neutral state. It should be noted that the operator’s yawn and closed eyes were classified as surprise or fear. It was significant to note that the emotional background of the operators is different. When choosing parameters for determining fatigue, an individual approach is required. Experiments also are being conducted to identify the correlation between blink parameters and piloting accuracy characteristics. It was also shown that the frequency of the operator blinking decreases due to the perception of new information and concentration.