Lottery Ticket Hypothesis: Placing the k-orrect Bets

摘要

Neural Network pruning has been one of the widely used techniques for reducing parameter count from an over-parametrized network. In the paper titled Deconstructing Lottery Tickets, the authors showed that pruning is a way of training, which we extend to show that pruning a well-trained network at initialization does not exhibit significant gains in accuracy. Stabilizing the Lottery Ticket Hypothesis motivates us to explore pruning after k~(th) epoch. We show that there exists a minimum value of k above which there is insignificant gain in accuracy, and the network enjoys a maximum level of pruning at this value of k while maintaining or increasing the accuracy of the original network. We test our claims on MNIST, CIFAR10, and CIFAR100 with small architectures such as lenet-300-100, Conv-2,4,6, and more extensive networks such as Resnet-20. We then discuss why pruning at initialization does not result in considerable benefits compared to pruning at k.