PROCESSING OF REDUCTION AND BROADCAST OPERATIONS ON LARGE DATASETS WITH MULTI-DIMENSIONAL HARDWARE ACCELERATORS

摘要

Methods, systems, and apparatus, including instructions encoded on storage media, for performing reduction of gradient vectors and similarly structured data that are generated in parallel, for example, on nodes organized in a mesh or torus topology defined by connections in at least two dimension between the nodes. The methods provide parallel computation and communication between nodes in the topology.