Superpixel based continuous conditional random field neural network for semantic segmentation

摘要

Recently, unifying conditional random fields (CRFs) into convolutional neural network (CNN) and enlarging spatial feature maps are two principal strategies for improving semantic segmentation performance. It is observed that building CRF defined on full resolution features into CNN can generate segmentation masks of better quality, however there exist several challenges related to training efficiency and computation costs. To design a more powerful segmentation model by using CRF and full resolution features, this paper proposes a novel fully supervised scheme for semantic segmentation based on superpixels and continuous CRF (C-CRF). Our new architecture includes three subnetworks: a unary network, a pairwise network and a superpixel based continuous CRF network (SCCN). The unary network upsamples low resolution features via de-convolutional structures, while pairwise network learns pixel-level pairwise similarities. The outputs of the two networks are fed to the SCCN that consists of two differentiable superpixel pooling layers and a differentiable C-CRF layer, leading to inference between superpixels in an image. The whole framework is trained in an end-to-end way by the jointly pixel-level and superpixel-level supervised learning strategy. Besides, C-CRF inference is fused with pixel-level prediction during the test procedure to decrease the effect of superpixel quality. In the experiments, we evaluate the power of the three subnetworks and the learning strategy comprehensively. Results on three benchmark datasets demonstrate that compared with baseline networks and other CRF based semantic segmentation methods, performance gain can be achieved by using the proposed SCCN. (C) 2019 Elsevier B.V. All rights reserved.