An improved mini-batching technique: Sample-and-Learn

摘要

Artificial neural networks suffer from prolonged training times, this is intensified when the volume of data is big. Mini-batching has become the standard in training neural networks. By reducing the number of data used to train each iteration the training time greatly shortens; even in a big data environment. A number of techniques such as parallel computation have rendered mini-batching a necessity for complex neural network models. Due to simplicity however, mini-batching methods have a number of inherent disadvantages that can affect the accuracy and convergence of a model. In this work, we focus on the ordering of samples presented to a neural network and propose a random sampling approach for generating mini-batches in linear time. Experimental results show that networks using our proposed Sample-and-Learn approach converge in fewer iterations while providing comparable or better accuracy.