MEgATrack: Monochrome Egocentric Articulated Hand-Tracking for Virtual Reality

摘要

mentedreality (VR/AR) experiences. Using four fisheye monochrome cameras,our system generates accurate and low-jitter 3D hand motion across alarge working volume for a diverse set of users.We achieve this by proposingneural network architectures for detecting hands and estimating hand keypointlocations. Our hand detection network robustly handles a variety ofreal world environments. The keypoint estimation network leverages trackinghistory to produce spatially and temporally consistent poses. We designscalable, semi-automated mechanisms to collect a large and diverse set ofground truth data using a combination of manual annotation and automatedtracking. Additionally, we introduce a detection-by-tracking method thatincreases smoothness while reducing the computational cost; the optimizedsystem runs at 60Hz on PC and 30Hz on a mobile processor. Together, thesecontributions yield a practical system for capturing a user’s hands and is thedefault feature on the Oculus Quest VR headset powering input and socialpresence.