Towards a deep human activity recognition approach based on video to image transformation with skeleton data

摘要

One of the most recent challenging tasks in computer vision is Human Activity Recognition (HAR), which aims to analyze and detect the human actions for the benefit of many fields such as video surveillance, behavior analysis and healthcare. Several works in the literature are based on the extraction and analysis of human skeletons in the aim of actions recognition. This paper introduces a new HAR approach based on the extraction of human skeletons from videos. Three features extraction techniques are proposed in this work. They used the extracted skeletons from the videos frames in order to construct a single image that summarizes the activity in that video. The first technique, called dynamic skeleton, is founded on the concept of dynamic images introduced in the literature, while the second one, called skeleton superposition, is based on the superposition of the extracted human skeletons in the same image. The third contribution is called body articulations and it uses only the body joints instead of the whole skeleton in order to recognize the ongoing activity. The obtained images from these three techniques are analyzed and classified using a classification system based on transfer learning principle by fine-tuning three well-known pre-trained CNNs (MobileNet, ResNet-50, VGG16). The designed system is validated and tested on two famous datasets for human activity recognition, which are RGBD-HuDact and KTH datasets. The obtained results are outstanding and proved that the implemented system outperforms the state-of-the-art approaches.