Precise Estimation of Reading Activities with Face Image and Read Aloud Voice

摘要

In Japanese public primary schools, every pupil may use an ICT device individually and simultaneously. In normal primary school, a few teachers must teach all pupils in a class. It is difficult to help all pupils to use an ICT device. For using ICT devices individually in a normal class, the ICT device help its' user by itself. To help the user, the ICT device must understand the state of the user. To help a teacher, it must precisely understand the users' reading activities. Reading ability is the base of all subjects. It is important that pupils acqure reading ability. This paper proposes a method to recognize the precise reading activity of a user with read aloud voices and facial images, shows its implementation, and experimental results. With the cooperative analysis of a read aloud voice from a microphone and a movement of a mouth from a camera, our implementation enables to estimate the action of read aloud much more precisely. The timing of a read aloud action is estimated in phrase by phrase manner. In-vitro experiments confirm the performance of our implementation.