Training very deep neural networks: Rethinking the role of skip connections

摘要

State-of-the-art deep neural networks (DNNs) typically consist of several layers of features representa-tions, and especially rely on skip connections to avoid the difficulty of model optimization. Despite the proliferation of different DNN models that employ various forms of skip connections to achieve remark-able results on benchmarking datasets, a concrete explanation for the successful operation and improved generalization capability of these DNNs is surprisingly still lacking. In this paper, we focus on investigat-ing the role of skip connections for training very deep DNNs. Our exposition directly provides interesting insights and new interpretations to the following important questions (i) why model optimization is easier (ii) why model generalization is better. Theoretical results reveal that skip connections allow DNNs to circumnavigate the singularity of latent representations that translate to optimization and gen-eralization problems, which plague models without skip connections referred to as PlainNets. For sub-stantiating our analysis, our investigation puts into context some of the most successful skip-connection based DNNs, which include residual networks (ResNets) and residual network with aggre-gated features (ResNeXt) in relation to PlainNets. Experimental evaluations of these models support the theoretical analysis.& nbsp; (c) 2021 Elsevier B.V. All rights reserved.