Disparity map estimation and view synthesis using temporally adaptive triangular meshes

摘要

This paper presents a new method for spatio-temporally coherent disparity map estimation and view interpolation for multiview linear camera arrays based on 2D domain triangulation. In the first frame of the sequence, a 3D mesh is computed for each camera, leading to a spatially coherent view interpolation. For the remaining frames of the sequence, a new scheme is proposed to update the 3D mesh dynamically, by moving, deleting and inserting vertices based on the optical flow and the previously computed disparity map. With this approach it is possible to relate triangles of the mesh across time, and a combination of Hidden Markov Models (HMMs) applied to time-persistent triangles with the Kalman Filter applied to the vertices produces temporally coherent disparity maps and interpolated views. Experimental results indicate that our approach was able to generate visually coherent in-between interpolated views for challenging, real-world videos with natural lighting and camera movement. Also, quantitative evaluations using objective video quality metrics show that our interpolated videos are typically better than competitive approaches. (C) 2017 Elsevier Ltd. All rights reserved.