A Quantitative Analysis on Required Network Bandwidth for Large-Scale Parallel Machine Learning

摘要

Parallelization is essential for machine learning systems that deals with large-scale dataset. Data parallel machine leaning systems that are composed of multiple machine learning modules, exchange the parameter to synchronize the models in the modules through network. We investigate the network bandwidth requirements for various parameter exchange method using a cluster simulator called SimGrid. We have confirmed that (1) direct exchange methods are substantially more efficient than parameter server based methods, and (2) with proper exchange methods, the bisection-bandwidth of network does not affect the efficiency, which implies smaller investment on network facility will be sufficient.