Investigation of Sign Language Motion Classification by Feature Extraction Using Keypoints Position of OpenPose

摘要

So far. on the premise of using a monocular optical camera, sign language motion classification has been performed using a wristband and color gloves with different dyeing on each finger. In this method, the movement of sign language is detected by extracting the color region using color gloves. However, this method has problems such as the burden on the signer of wearing color gloves and the change in color extraction accuracy due to changes in ambient light, resulting in difficulty in ensuring stable classification accuracy. Therefore, we used OpenPose, which can detect the movements of both hands without the need for colored gloves, to classify sign language movements. Feature element extraction was performed using the keypoint position obtained from OpenPose. Then, we proposed three methods as feature element for classifying each motion and compared their classification accuracy. In method 1. feature element is obtained directly from the keypoint positions of the neck, shoulder, elbow, and wrist. Method 2 is a scheme of obtaining from the relative distance from the target keypoint position around the neck. In method 3. the feature element was 30 elements, which is the sum of the 24 elements obtained in method 1 and the 6 elements obtained in method 2. In the classification experiment, cross-validation was performed using the feature quantity obtained from the sign language motion videos of five people, and the accuracy of each method was investigated. In method 1, B (68.05%), A (62.56%), C (62.19%), D (61.49%), E (56.75%). average 62.21%, in order from the signer with the highest average classification accuracy. In method 2, B (75.31%), A (75.09%), D (73.28%), E (69.97%), C (69.81%), average 72.69%. Method 3 gave B (70.72%), A (69.65%), C (66.13%), D (64.27%), E (62.72%), and an average of 66.30% classification accuracy.