Accelerating the Forward Convnets with Software Pipeline

摘要

We present a new approach to accelerate the forward convolutional neural netWorks (ConvNets). Instead of taking data parallelism into account for faster inference, We speed up ConvNets With the technology of software pipeline. Since a ConvNet is a chain-like model, Which means that the output of current layer Would become the input of next layer, it is natural to accelerate it With the use of software pipeline. Firstly, the forward propagation in each layer, treated as a minimum unit, is timed during the inference. Then, according to the computation time in each layer, one or several units Would compose a working part With an allocation algorithm. Finally, these Working parts Would be executed in different computing resource simultaneously, so that the ConvNet could be operated by software pipeline. This is a conceptually simple, flexible and general Way to accelerate the forward ConvNet, We named it pipeline ConvNet (PipeCNN). Experiments show that the method could obtain good speedup and improve the throughput of the networks.