Evaluating the Potential Gain of Auditory and Audiovisual Speech-Predictive Coding Using Deep Learning

摘要

Sensory processing is increasingly conceived in a predictive frameworkin which neurons would constantly process the error signal resultingfrom the comparison of expected and observed stimuli. Surprisingly,few data exist on the accuracy of predictions that can be computedin real sensory scenes. Here, we focus on the sensory processing ofauditory and audiovisual speech. We propose a set of computationalmodels based on artificial neural networks (mixing deep feedforwardand convolutional networks), which are trained to predict future audioobservations from present and past audio or audiovisual observations(i.e., including lip movements). Those predictions exploit purelylocal phonetic regularities with no explicit call to higher linguistic levels.Experiments are conducted on the multispeaker LibriSpeech audiospeech database (around 100 hours) and on the NTCD-TIMIT audiovisualspeech database (around 7 hours). They appear to be efficient in ashort temporal range (25–50 ms), predicting 50% to 75% of the varianceof the incoming stimulus, which could result in potentially saving upto three-quarters of the processing power. Then they quickly decreaseand almost vanish after 250 ms. Adding information on the lips slightlyimproves predictions, with a 5% to 10% increase in explained variance.Interestingly the visual gain vanishes more slowly, and the gain is maximumfor a delay of 75 ms between image and predicted sound.