FaceAUG: A Cross-Platform Application for Real-Time Face Augmentation in Web Browser

摘要

This paper presents FaceAUG, a cross-platform application for real-time face augmentation in a web browser. Human faces are detected and tracked in real-time from the video stream of the embedded or separated webcam of the user device. Then, the application overlays different 2D or 3D augmented reality (AR) filters and effects over the region of the detected face(s) to achieve a mixed virtual and AR effect. A 2D effect can be a photo frame or a 2D face mask using an image from the local repository. A 3D effect is a 3D face model with a colored material, an image texture, or a video texture. The application uses TensorFlow.js to load the pre-trained Face Mesh model for predicting the regions and landmarks of the faces that appear in the video stream. Three.js is used to create the face geometries and render them using the material and texture selected by the user. FaceAUG can be used on any device, as long as an internal or external camera and a state-of-the-art web browser are accessible on the device. The application is implemented using front-end techniques and is therefore functional without any server-side supports at back-end. Experimental results on different platforms verified the effectiveness of the proposed approach.