Social game retrieval from unstructured videos.

摘要

Parent-child social games, such as peek-a-boo and patty-cake, are a key element of an infant's earliest social interactions. The analysis of children's behaviors in social games based on video recordings provides a means for psychologists to study their social and cognitive development. However, the current practice in the use of video for behavioral research is extremely labor-intensive, involving many hours spent extracting and coding relevant video clips from a large corpus. From the standpoint of computer vision, such real-world video collections pose significant challenges in the automatic analysis of behavior, such as cluttered backgrounds, the effect of varying camera angles, clothing, subject appearance and lighting. These observations motivate my thesis work---automatic retrieval of social games from unstructured videos. The goal of this work is both to help accelerate the research progress in behavioral science and to take the initial steps towards the analysis of natural human interactions in natural settings.;Social games are characterized by repetitions of turn-taking interactions between the parent and the child, with variations that are recognizable by both of them. I developed a computational model for social games that exploits the temporal structure over a long time-scale window as quasi-periodic patterns in a time series. I presented an unsupervised algorithm that mines the quasi-periodic patterns from videos. The algorithm consists of two functional modules: converting image sequences into discrete symbolic sequences and mining quasi-periodic patterns from the symbolic sequences. When this technique is applied to video of social games, the extracted quasi-periodic patterns often correspond to meaningful stages of the games. The retrieval performance on unstructured, lab-recorded videos and real-world family movies is promising. Building on this work, I developed a new feature extraction algorithm for social game categorization. Given a quasi-periodic pattern representation, my method automatically selects the most relevant space-time interest points to construct the feature representation. Our experiments demonstrate very promising classification performance on social games collected from YouTube. In addition, the method can also be used to categorize TV videos of sports rallies, demonstrating the generality of this approach. In order to support and encourage more research on human behavior analysis in realistic contexts, a video database of realistic child play in natural settings has been collected and is published on our project website (http://www.cc.gatech.edu/cpl/projects/socialgames), along with annotations.;The unsupervised quasi-periodic pattern mining method represents a substantial generalization of conventional periodic motion analysis. Its generality is evaluated by retrieving motions of a range of quasi-periodicity from unstructured videos. The performance was compared with that of a periodic motion detection method based on motion self-similarity. Our method demonstrates superior retrieval performance with a 100% precision when the recall is up to 92.04%, with much fewer parameters than that of the other method.