SPECULATIVE TRAINING USING PARTIAL GRADIENT UPDATE

摘要

The exchange of weighting gradients between the processing nodes can lead to a considerable bottleneck in the training process. Instead of staying inactive during the weight gradient exchange process, a processing node can update its own set of weights for the next iteration of the training process using the processing node's local weight gradients. The next iteration of training can be started using these speculative weights until the weight gradient exchange process is complete and a global weight update is available. If the speculative weights are close enough to the weight values from the global weight update, the training process at the processing node can continue training using the results calculated from the speculative weights to reduce the overall training time.