Realtime 3D Eye Gaze Animation Using a Single RGB Camera

摘要

This paper presents the first realtime 3D eye gaze capture methodrnthat simultaneously captures the coordinated movement of 3D eyerngaze, head poses and facial expression deformation using a singlernRGB camera. Our key idea is to complement a realtime 3D facialrnperformance capture system with an efficient 3D eye gaze tracker.rnWe start the process by automatically detecting important 2D facialrnfeatures for each frame. The detected facial features are then usedrnto reconstruct 3D head poses and large-scale facial deformation usingrnmulti-linear expression deformation models. Next, we introducerna novel user-independent classification method for extractingrniris and pupil pixels in each frame. We formulate the 3D eye gazerntracker in the Maximum A Posterior (MAP) framework, which sequentiallyrninfers the most probable state of 3D eye gaze at eachrnframe. The eye gaze tracker could fail when eye blinking occurs.rnWe further introduce an efficient eye close detector to improve thernrobustness and accuracy of the eye gaze tracker. We have tested ourrnsystem on both live video streams and the Internet videos, demonstratingrnits accuracy and robustness under a variety of uncontrolledrnlighting conditions and overcoming significant differences of races,rngenders, shapes, poses and expressions across individuals.