The algorithms for the acquisition of linguistic speech units based on integrating perceptual information

摘要

This report describes a theory and algorithm for acquisition of words and speech segments, which compose a word, by integrating speech and image information. Main argument is how to acquire a hierarchical structure of speech units by using image information combined with speech. An information theoretical principle of learning algorithm based on maximum mutual information criteria is proposed as a solution to this argument. Speech segments, the word consisting of speech segments and image concepts corresponding to each word are acquired simultaneously by applying proposed learning method. In the algorithm, speech segments, words, and dynamic image concepts (motions) are represented by Hidden Markov Models, and static image concepts are represented by multivariate normal distribution. The number of these stochastic models and the values of model parameters are automatically determined. The utterances of isolated words and the images of stuffed toys are used for the learning data in experiments. The experimental results show that linguistic speech units and image concepts were adequately acquired.