Modulation of Activation Function in Triangular Recurrent Neural Networks for Time Series Modeling

摘要

This paper introduces a novel method to dynamically vary the activation function slopes in recently developed upper and lower triangular recurrent neural networks (ULTRNN) to enhance their modeling capability. The ULTRNN employs a pair of triangular feedback weight matrices with block diagonal elements whose eigenvalues are constrained to lie on or close to the unit circle in the complex z-plane to maintain network and training stability. The activation function slopes of the ULTRNN state variables are dynamically varied by a second modulating network. The inputs to the modulating network are the state variables of the principal ULTRNNs and their inputs. The modulating network is trained simultaneously with the principal ULTRNN to compute the activation function slope for the latter’s each state variable at each time step. Such dynamic variation of the activation function slopes selectively enhances the contribution of certain states while suppressing that of the others. A larger slope results in a longer time contribution of the corresponding state and helps model long-term dependencies. Conversely, a smaller slope results in a shorter time contribution and may be used to model controlled "forgetting". The proposed modulation technique enhances the ULTRNN’s ability to effectively incorporate short-term memory and long-term dependencies. Simulation results show that with activation function modulation the ULTRNNs are able to autonomously replicate the outputs of sample chaotic dynamic system with good accuracy. This capability can be highly effective in modeling or characterizing the inherent process that generates the time series.