SYSTEM FOR EFFICIENT LARGE-SCALE DATA DISTRIBUTION IN DISTRIBUTED AND PARALLEL PROCESSING ENVIRONMENT

摘要

The present invention relates to a system for efficient large-scale data distribution in a distributed and parallel processing environment. In particular, the present invention relates to global Top-k sparsification for low bandwidth networks. The present invention verifies that gTop-k S-SGD has nearly consistent convergence performance with S-SGD and evaluates the training efficiency of gTop-k on a cluster with 32 GPU machines which are inter-connected with 1 Gbps Ethernet. The experimental results show that the present invention achieves up to 2.7-12× higher scaling efficiency than S-SGD with dense gradients, and 1.1-1.7× improvement than the existing Top-k S-SGD.