We propose BinaryRelax, a simple two-phase algorithm, for training deepneural networks with quantized weights. The set constraint that characterizesthe quantization of weights is not imposed until the late stage of training,and a sequence of pseudo quantized weights is maintained. Specifically, werelax the hard constraint into a continuous regularizer via Moreau envelope,which turns out to be the squared Euclidean distance to the set of quantizedweights. The pseudo quantized weights are obtained by linearly interpolatingbetween the float weights and their quantizations. A continuation strategy isadopted to push the weights towards the quantized state by gradually increasingthe regularization parameter. In the second phase, exact quantization schemewith a small learning rate is invoked to guarantee fully quantized weights. Wetest BinaryRelax on the benchmark CIFAR-10 and CIFAR-100 color image datasetsto demonstrate the superiority of the relaxed quantization approach and theimproved accuracy over the state-of-the-art training methods. Finally, we provethe convergence of BinaryRelax under an approximate orthogonality condition.
展开▼
