PARALLEL COMPUTING SCHEME GENERATION FOR NEURAL NETWORKS

摘要

Various embodiments relate to determining a parallel computation scheme for a neural network. A device may receive a computation graph and transform the computation graph into a dataflow graph comprising recursive subgraphs. Each recursive subgraph may comprise a tuple of another recursive subgraph and an operator node, or an empty graph. The device may determine a number of partitioning recursions based on a number of parallel computing devices. For each partitioning recursion, the device may determine costs corresponding to operator nodes, determine a processing order of the recursive subgraphs, and process the recursive subgraphs. To process a recursive subgraph, the device may select a partitioning axis for tensor(s) associated with an operator node of the recursive subgraph. The device may output a partitioning scheme comprising partitioning axes for each tensor associated with the operator nodes. Devices, methods, and computer programs are disclosed.