DISTRIBUTED DEEP LEARNING DEVICE AND DISTRIBUTED DEEP LEARNING SYSTEM

摘要

A distributed deep learning device that exchanges a quantized gradient with a plurality of learning devices and performs distributed deep learning, that includes: a communicator that exchanges the quantized gradient by communication with another learning device; a gradient calculator that calculates a gradient of a current parameter; a quantization remainder adder that adds, to the gradient, a value obtained by multiplying a remainder at the time of quantizing a previous gradient by a predetermined multiplying factor; a gradient quantizer that quantizes the gradient obtained by the quantization remainder adder; a gradient restorer that restores a quantized gradient received by the communicator to a gradient of the original accuracy; a quantization remainder storage that stores a remainder at the time of quantizing; a gradient aggregator that aggregates gradients collected by the communicator and calculates an aggregated gradient; and a parameter updater that updates the parameter with the aggregated gradient.