Analysis of Gradient Degradation and Feature Map Quality in Deep All-Convolutional Neural Networks Compared to Deep Residual Networks

摘要

The introduction of skip connections used for summing feature maps in deep residual networks (ResNets) were crucially important for overcoming gradient degradation in very deep convolutional neural networks (CNNs). Due to the strong results of ResNets, it is a natural choice to use features that it produces at various layers in transfer learning or for other feature extraction tasks. In order to analyse how the gradient degradation problem is solved by ResNets, we empirically investigate how discriminability changes as inputs propagate through the intermediate layers of two CNN variants: all-convolutional CNNs and ResNets. We found that the feature maps produced by residual-sum layers exhibit increasing discriminability with layer-distance from the input, but that feature maps produced by convolutional layers do not. We also studied how discriminability varies with training duration and the placement of convolutional layers. Our method suggests a way to determine whether adding extra layers will improve performance and show how gradient degradation impacts on which layers contribute increased discriminability.