System and method for deep learning based hand gesture recognition in first person view

摘要

A system and method for hand-gesture recognition are provided. The method includes receiving frames of a media stream of a scene captured from a first person view (FPV) of a user using at least one RGB sensor communicably coupled to a wearable AR device. The media stream includes RGB image data associated with the frames of the scene. The scene comprises a dynamic hand gesture performed by the user. A temporal information associated with the dynamic hand gesture is estimated from the RGB image data by using a deep learning model. The estimated temporal information is associated with hand poses of the user and comprising a plurality of key-points identified on user's hand in the plurality of frames. Based on the temporal information of the key points, the dynamic hand gesture is classified into at least one predefined gesture class by using a multi-layered LSTM classification network.