Joint Depth estimation and semantic segmentation from a single image

摘要

Global semantic and depth layouts of a scene in an image are estimated (126) through machine learning (e.g. as a template classification problem) by computing device(s). Local semantic and depth layouts are also estimated (128) for respective ones of a plurality of segments of the scene of the image through machine learning (e.g. using a convolutional neural network, CNN, or support vector machine, SVM) by the computing device(s) (e.g. also as a template classification problem). The estimated global semantic and depth layouts are merged (130) (e.g. using conditional random field) with the local semantic and depth layouts by the computing device(s) to semantically label (120) and assign a depth value (122) to image pixels (e.g. thereby ensuring consistency). The estimated global depth layout may assign a respective absolute distance to a plurality of pixels in the image. The merging may include smoothing the depth labels assigned to individual pixels. Also disclosed is the prediction of local semantic and depth layouts of individual segments using the estimated global semantic and depth layouts of the scene, and jointly forming a semantically-labelled version of the image in which individual pixels are assigned a semantic label and depth value.