Towards Perceptually Realistic Talking Heads: Models, Methods and McGurk

摘要

Motivated by the need for an informative, unbiased and quantitative perceptual method for the development and evaluation of a talking head we are developing, we propose a new test based on the "McGurk Effect". Our approach helps to identify strengths and weaknesses in underlying talking head algorithms, and uses this insight to guide further development. The test also evaluates the realism of talking head behavior in comparison to real speaker footage, painting an overall picture of a talking head's performance. By distracting a participant's attention away from the true nature of the test, we also obtain an unbiased view on talking head performance - since the participant's prior concerning what is synthetic animation and what is real footage is not encouraged to develop. Our current talking head is a hierarchical 2D image based model, trained from real speaker video footage and continuous speech signals. After training, the talking head may be animated using new continuous speech signals not previously encountered in the training set, and produces realistic lip-synched animations. We apply our McGurk perceptual test to our model and demonstrate how we are able to evaluate and identify some of its strengths and weaknesses. We then suggest how our underlying algorithm may be improved in light of the evaluation.