Accurate image registration through three-dimensional reconstruction.

摘要

Image registration is a fundamental problem in image analysis, and is used in a variety of applications such as tracking, moving object detection, remote sensing, etc. In this thesis, we study image registration problems where the images are taken at different times, from different sensors, and from different viewpoints. Moreover, the scene may contain arbitrary 3D structure. With all these variations, image registration become a challenging task, where standard techniques may fail to produce accurate results.;According to the geometry in the image, we categorize image registration problems into 2D registration and 3D registration. In 2D registration, image features are extracted and matched to establish correspondences, from which the epipolar geometry can be estimated. Images are then registered using a derived 2D model. In 3D registration, on top of epipolar geometry estimation, a sparse reconstruction step is required which recovers camera parameters and the sparse structure. A dense reconstruction step follows, which recovers structure in the entire scene, images are then registered through 3D inference. The main contribution in this thesis is the in depth study of a number of issues in image registration applications which a general framework does not address, particularly in the 3D reconstruction pipeline.;We start from 2D image registration, and look at two applications, UAV image registration and retinal image registration. In UAV image registration, we are given a UAV image sequence, and the goal is to produce a mosaic in a progressive manner. As inter-frame registration error accumulates along the process, and results in deviation, we introduce an additional map as a global reference, and perform UAV to map registration to compensate for the error. In retinal image registration, the input is a set of retinal images in multiple modalities. We propose an iterative nearest neighbor matching method to account for issues raised in multi-modal imagery, and achieve both high registration rate and high efficiency.;We then extend our study in both imageries to 3D, to account for the underlying 3D geometry. In addition, we research some other 3D reconstruction problems using human facial images. In 3D retinal image registration, we address the issues which arise from the near planar property of a retinal surface, and propose a 4-pass bundle adjustment method to account for it. Our approach is shown to be very robust and efficient, and is state-of-the-art in 3D retinal image registration. For UAV image registration, we focus on the dense 3D reconstruction of urban environments. Images of urban environments are characterized by significant occlusions, sharp edges, and textureless regions, leading to poor 3D reconstruction using standard multi-view stereo algorithms. Our approach makes a general assumption that urban scenes consist of planar facets that are either horizontal or vertical. These two assumptions provide very strong constraints for the underlying geometry. The contribution of this work is the way we translate these constraints respectively into intra-image-column and inter-image-column constraints, and formulate the dense reconstruction problem as a 2-pass dynamic programming problem, which can be solved efficiently. Moreover, our algorithm is fully parallelizable, which is appropriate for GPU computing. Our results show that we can preserve a high level of detail, and have high visual quality. In 3D human face reconstruction, we are given a set of 5 wide-baseline images that are only weakly calibrated. The focus in this work is on both sparse reconstruction and dense reconstruction. First, to calibrate cameras, we propose an iterative bundle adjustment approach to solve the challenging wide-baseline feature matching problem. Then, for dense reconstruction, we propose to use a face-specific cylindrical representation which allows us to solve a global optimization problem for N-view dense aggregation. We explicitly use profile contours extracted from the image in both sparse reconstruction and dense reconstruction steps to provide strong constraints for the underlying geometry. Experimental results show that our method provides accurate and stable reconstruction results on wide-baseline images. We compare our method with state-of-the-art methods, and show that it provides significantly better results in terms of both accuracy and efficiency.