XNect: Real-time Multi-Person 3D Motion Capture with a Single RGB Camera

摘要

We present a real-time approach for multi-person 3D motion capture at over30 fps using a single RGB camera. It operates successfully in generic sceneswhich may contain occlusions by objects and by other people. Our method operatesin subsequent stages. The first stage is a convolutional neural network(CNN) that estimates 2D and 3D pose features along with identity assignmentsfor all visible joints of all individuals.We contribute a new architecture for this CNN, called SelecSLS Net, that uses novel selective long and short rangeskip connections to improve the information flow allowing for a drasticallyfaster network without compromising accuracy. In the second stage, a fullyconnectedneural network turns the possibly partial (on account of occlusion)2Dpose and3Dpose features for each subject into a complete3Dpose estimateper individual. The third stage applies space-time skeletal model fitting to thepredicted 2D and 3D pose per subject to further reconcile the 2D and 3D pose,and enforce temporal coherence. Our method returns the full skeletal pose injoint angles for each subject. This is a further key distinction from previouswork that do not produce joint angle results of a coherent skeleton in real timefor multi-person scenes. The proposed system runs on consumer hardware ata previously unseen speed of more than 30 fps given 512x320 images as inputwhile achieving state-of-the-art accuracy, which we will demonstrate on arange of challenging real-world scenes.