ARL-TR-8580 - Scaling to Multiple Graphics Processing Units (GPUs) in TensorFlow | U.S. Army Research Laboratory

摘要

Although accuracies of neural networks are surpassing human performance, training a deep neural network is a timeconsumingtask due to its increasing high-dimensional parameters. It is not uncommon for the training of deep neuralnetworks to run for a week. Accordingly, the size of neural networks has doubled every 2.4 years, exhibiting an exponentialgrowth from 1958 to 2014. The increasing size of neural network architectures will likely lead to higher computationalcomplexity that will need scalable solutions. To mitigate the computational requirement and maximize throughput, this workfocuses on multi-graphics-processing-unit scalability.