From talking head to singing head: A significant enhancement for more natural human computer interaction

摘要

This paper proposes a 3D virtual animating head system, which can not only talk but also sing. With a reconstructed head mesh model, including external/internal articulators, from multi-source images, biology information are first used to visualize each phoneme with a musical note. The synchronicity between songs and articulatory movements is then modeled by a deep neural network trained on an audio/articulatory corpus. Finally, the visualization results of phonemes are blended by the synchronicity model to produce the song synchronized articulatory animations. Quantitative and qualitative improvements of singing ability on human computer interaction are demonstrated by comparing with other state-of-the-art talking head systems.