Switch Unit (SU): A Novel Type of Unit for the Activation Function

摘要

The paper presents a novel unit - switch unit (SU). This unit represents a conditional include function, which is able to map the input from one dimensional space to two dimensional spaces depending on whether the input is positive or not. In the traditional neural networks, there always exist one of derivatives of two entries from output that refers to input as one and the other as zero. The proposed unit is able to cope with the vanishing gradient issues induced by the factor of activation function such as ReLUs (rectified linear units). In the experiments, different neural networks are trained based on the proposed units and tested on CIFAR-10 and CIFAR-100 datasets, which yielded the comparable results to ReLUs based neural networks using the same parameters.