Efficient Learning Algorithm Using Compact Data Representation in Neural Networks

摘要

Convolutional neural networks have dramatically improved the prediction accuracy in a wide range of applications, such as vision recognition and natural language processing. However the recent neural networks often require several hundred megabytes of memory for the network parameters, which in turn consume a large amount of energy during computation. In order to achieve better energy efficiency, this work investigates the effects of compact data representation on memory saving for network parameters in artificial neural networks while maintaining comparable accuracy in both training and inference phases. We have studied the dependence of prediction accuracy on the total number of bits for fixed point data representation, using a proper range for synaptic weights. We have also proposed a dictionary based architecture that utilizes a limited number of floating-point entries for all the synaptic weights, with proper initialization and scaling factors to minimize the approximation error. Our experiments using a 5-layer convolutional neural network on Cifar-10 dataset have shown that 8 bits are enough for bit width reduction and dictionary based architecture to achieve 96.0% and 96.5% relative accuracy respectively, compared to the conventional 32-bit floating point.