PROCESSING SYMBOLIC SEQUENCES USINGECHO-STATE NETWORKS

摘要

A novel recurrent neural network (RNN) model, called the echo state network (ESN), wassuccessfully applied in several time series processing tasks. Also using ESNs forprocessing symbolic sequences seems to be a promising possibility. But ESNs, like RNNsinitialized with small weights, share some properties with variable length Markov modelswhen used for processing symbolic sequences. We call this phenomenon a Markovianarchitectural bias since meaningful and potentially a useful Markov-like state spaceorganization is present in an RNN prior to any training. In this paper we first explain thisnotion in detail. Then, in an experimental section, we compare the performance of ESNswith connectionist models explicitly using the Markovian architectural bias property andwith variable length Markov models (VLMMs). We show that ESNs performanceremains almost the same for a wide range of parameters and that the number of reservoirunits plays a similar role to the number of context units of other models. We show thatESNs, like other connectionist models with state space organized according to theMarkovian architectural bias property, cannot generalize well on subsequences notpresent in the training set.