Mask-off: Synthesizing Face Images in the Presence of Head-mounted Displays

摘要

A head-mounted display (HMD) could be an important component of augmentedreality system. However, as the upper face region is seriously occluded by thedevice, the user experience could be affected in applications such astelecommunication and multi-player video games. In this paper, we first presenta novel experimental setup that consists of two near-infrared (NIR) cameras topoint to the eye regions and one visible-light RGB camera to capture thevisible face region. The main purpose of this paper is to synthesize realisticface images without occlusions based on the images captured by these cameras.To this end, we propose a novel synthesis framework that contains four modules:3D head reconstruction, face alignment and tracking, face synthesis, and eyesynthesis. In face synthesis, we propose a novel algorithm that can robustlyalign and track a personalized 3D head model given a face that is severelyoccluded by the HMD. In eye synthesis, in order to generate accurate eyemovements and dynamic wrinkle variations around eye regions, we propose anothernovel algorithm to colorize the NIR eye images and further remove the "red eye"effects caused by the colorization. Results show that both hardware setup andsystem framework are robust to synthesize realistic face images in videosequences.