EgoCap: Egocentric Marker-less Motion Capture with Two Fisheye Cameras

摘要

Marker-based and marker-less optical skeletal motion-capture methodsrnuse an outside-in arrangement of cameras placed around a scene,rnwith viewpoints converging on the center. They often create discomfortrnwith marker suits, and their recording volume is severelyrnrestricted and often constrained to indoor scenes with controlledrnbackgrounds. Alternative suit-based systems use several inertialrnmeasurement units or an exoskeleton to capture motion with anrninside-in setup, i.e. without external sensors. This makes capturernindependent of a confined volume, but requires substantial, oftenrnconstraining, and hard to set up body instrumentation. Therefore,rnwe propose a new method for real-time, marker-less, and egocentricrnmotion capture: estimating the full-body skeleton pose from arnlightweight stereo pair of fisheye cameras attached to a helmet orrnvirtual reality headset – an optical inside-in method, so to speak.rnThis allows full-body motion capture in general indoor and outdoorrnscenes, including crowded scenes with many people nearby, whichrnenables reconstruction in larger-scale activities. Our approach combinesrnthe strength of a new generative pose estimation framework forrnfisheye views with a ConvNet-based body-part detector trained on arnlarge new dataset. It is particularly useful in virtual reality to freelyrnroam and interact, while seeing the fully motion-captured virtualrnbody.