In this work we present In-Place Activated Batch Normalization (InPlace-ABN)- a novel approach to drastically reduce the training memory footprint ofmodern deep neural networks in a computationally efficient way. Our solutionsubstitutes the conventionally used succession of BatchNorm + Activation layerswith a single plugin layer, hence avoiding invasive framework surgery whileproviding straightforward applicability for existing deep learning frameworks.We obtain memory savings of up to 50% by dropping intermediate results and byrecovering required information during the backward pass through the inversionof stored forward results, with only minor increase (0.8-2%) in computationtime. Also, we demonstrate how frequently used checkpointing approaches can bemade computationally as efficient as InPlace-ABN. In our experiments on imageclassification, we demonstrate on-par results on ImageNet-1k withstate-of-the-art approaches. On the memory-demanding task of semanticsegmentation, we report results for COCO-Stuff, Cityscapes and MapillaryVistas, obtaining new state-of-the-art results on the latter without additionaltraining data but in a single-scale and -model scenario. Code can be found athttps://github.com/mapillary/inplace_abn .
展开▼
