RECURRENT NETWORKS WITH MOTION-BASED ATTENTION FOR VIDEO UNDERSTANDING

摘要

A method of predicting action labels for a video stream includes receiving the video stream and calculating an optical flow of consecutive frames of the video stream. An attention map is generated from the current frame of the video stream and the calculated optical flow. An action label is predicted for the current frame based on the optical flow, a previous hidden state and the attention map.