Parallel processing of reduction and broadcast operations on large datasets of non-scalar data

摘要

Methods, systems, and apparatus, including instructions encoded on storage media, for performing reduction of gradient vectors and similarly structured data that are generated in parallel, for example, on nodes organized in a mesh or torus topology defined by connections in at least two dimensions between the nodes. The methods provide parallel computation and communication between nodes in the topology.