首页>
外国专利>
EFFICIENT PARALLEL TRAINING OF A NETWORK MODEL ON MULTIPLE GRAPHICS PROCESSING UNITS
EFFICIENT PARALLEL TRAINING OF A NETWORK MODEL ON MULTIPLE GRAPHICS PROCESSING UNITS
展开▼
机译:多图形处理单元上的网络模型的有效并行训练
展开▼
页面导航
摘要
著录项
相似文献
摘要
A system and method provides efficient parallel training of a neural network model on multiple graphics processing units. A training module reduces the time and communication overhead of gradient accumulation and parameter updating of the network model in a neural network by overlapping processes in an advantageous way. In a described embodiment, a training module overlaps backpropagation, gradient transfer and accumulation in a Synchronous Stochastic Gradient Decent algorithm on a convolution neural network. The training module collects gradients of multiple layers during backpropagation of training from a plurality of graphics processing units (GPUs), accumulates the gradients on at least one processor and then delivers the gradients of the layers to the plurality of GPUs during the backpropagation of the training. The whole model parameters can then be updated on the GPUs after receipt of the gradient of the last layer.
展开▼