System and method for deep learning based hand gesture recognition in first person view

摘要

To provide a system and a method for hand gesture recognition.SOLUTION: A method includes receiving, using at least one RGB sensor communicatively connected to a wearable AR device, a plurality of frames of a media stream of a scene obtained from a first person view (FPV) of a user. A scene includes a dynamic hand gesture performed by the user. Timing information associated with a dynamic hand gestures is estimated, using a deep learning model, from an RGB image data. The estimated timing information is associated with a hand pose by the user and further includes a plurality of key points specified on a hand of the user in a plurality of frames. The dynamic hand gesture is classified, using a multi-layer LSTM classification network, into at least one predefined gesture based on the timing information of the plurality of key points.SELECTED DRAWING: Figure 4