Joint space-time motion-based video segmentation and occlusion detection using multiphase level sets

摘要

Spatial video segmentation is usually performed based on motion between two frames. Some recent approaches extend this to joint segmentation in space-time; the resulting 3-D segmentation can be interpreted as a volume "carved out" by a moving object in the image sequence domain, or the so-called,"object tunnel". In this paper, we extend this concept to explicit modeling of occlusion events in space-time. In addition to the modeling of object evolution, we also explicitly model occluded and newly-exposed areas in the background by means of "occlusion volume", a new space-time concept. A voxel belongs to occlusion volume if its intensity is consistent with past intensities along its motion trajectory but inconsistent with future intensities (reversed for "exposed volume"). We propose a variational formulation of the problem that we solve using the multiphase level set method. We show encouraging experimental results for synthetic and natural image sequences.