D~3-LND: A two-stream framework with discriminant deep descriptor, linear CMDT and nonlinear KCMDT descriptors for action recognition

摘要

In order to improve recognition accuracy, a two-stream framework which incorporates deep-learned stream and hand-crafted stream is proposed. Firstly, a discriminant nonlinear feature fusion method is proposed, which introduces the category structure information and obtains the nonlinear relationships between features. Secondly, the global features and local features are extracted from deep network, both of them are fused by the proposed fusion method to obtain a discriminant deep descriptor. Thirdly, to capture the spatio-temporal characteristics of video, the temporal derivatives of gradient, optical flow and motion boundary are extracted in the space-time cube centered at trajectory and taken as low-level features. Subsequently, the covariance and kernelized covariance of low-level features are respectively computed to obtain the Covariance Matrix based on Dense Trajectory (CMDT) and Kernelized Covariance Matrix based on Dense Trajectory (KCMDT) descriptors. Finally, a two-stream framework with discriminant deep descriptor, linear CMDT and nonlinear KCMDT descriptors (D-3-LND) is presented, which shares the benefits of both deep-learned and hand-crafted features, and further improves recognition accuracy. Experiments on challenging HMDB51 and UCF101 datasets verify the effectiveness of our method. (C) 2018 Elsevier B.V. All rights reserved.