Hebbian Versus Gradient Training of ESN Actors in Closed-Loop ACD

摘要

The present work continues investigations on combination between Adaptive Critic Design (ACD) approach - a gradient-based optimization technique - and a more biologically plausible associative or Hebbian learning. Echo state network (ESN) was used as adaptive critic element that was trained minimizing temporal difference error. While in the previous work the actor was a time profile of the action variable, here investigations are extended to the closed loop (feedback) control scheme. The actor is another ESN network and its inputs are some of the process state variables while its output is the value of the controlled variable. The only trainable connections of the actor - from its reservoir to the readout - are trained to minimize (maximize) the critic output. Comparison between backpropagation of utility approach that is gradient descent algorithm and a Hebbian learning law is made. These two approaches are tested on a task for optimization of a complex nonlinear process for bio-polymer production. The obtained results are compared with respect to the convergence speed as well as to the obtained solution, i.e. reached local optima.