Low-Cost and Fast Design of Precise Activation Functions in Neural Network

摘要

Sigmoidal activation functions play a key role in AI accelerators. They have been usually implemented by hyperbolic sine and cosine functions with a division since the CORDIC algorithm was proposed in 1956 and unified in 1971. From all literature it seems that people have never tried the simple algorithm proposed in this paper, in which only a half-domain exponential function is applied prior to a Booth division. More than one half of iterations and area can be reduced. From experimental results, both the area overhead and the power consumption can be reduced to only 1/3. The acceleration in backpropagation and learning rate can then be highly improved.