Natural Interaction with 3D Content on Mobile AR Systems Using Gesture Recognition

摘要

We present a mobile AR interaction system where the user can naturally interact with and manipulate 3D content by recognizing discrete in-air gestures and temporal poses of a hand in front of the camera. Our system consists of established image processing, pose estimation and AR rendering steps, and a novel fingertip detection algorithm that can run real-time on off-the-shelf mobile devices with the use of an external depth camera. We also present an application prototype implemented on a tablet with a mounted IR structured-lighting depth camera.