In this paper, we describe a recent research results about how to generate an avatar's face on a realtime process exactly copying a real person's face. It is very important for synthesis of a real avatar to duplicate emotion and impression precisely included in original face image and voice. Face fitting tool from multi-angle camera images is introduced to make a real 3D face model with real texture and geometry very close to the original. When avatar is speaking something, voice signal is very essential to decide a mouth shape feature. So real-time mouth shape control mechanism is proposed by conversion from speech parameters to lip shape parameters using multi-layer neural network. For dynamic modeling of facial expression, muscle structure constraint is introduced to generate a facial expression naturally with a few parameters. We also tried to get muscle parameters automatically to decide an expression from local motion vector on face calculated by optical flow in video sequence. We also tried to control this artificial muscle model directly by EMG signal. To get more reality, modeling method of hair is also introduced and dynamics of hair in stream of wind can be achieved with low calculation cost. By using these several kinds of multi-modal signal sources, very natural face image and its impression can be duplicated on avatar's face.
展开▼
